No need to feel like a douche, you did reply, albeit quietly. I'm sure cab drivers have to deal with genuine douches all the time. I sometimes do the same, though rather than speaking quietly I take too long (silent block) and then think "it's a bit weird for me to say 'you too' now" and so just give up. Its something I am trying to correct.