Write your address in your phone notes and just show the driver the address without saying it. There shouldn’t be any problem with that.