I think what you need to do is tell him that you stutter sometimes so then there is no confusion there. Then tell him that when you are in close traffic, such as on city streets, that you want to stay silent so you’re not focusing on your speech and driving at the same time. You will sound like the man. Being proactive will make you come off so confident.