If your decision not to call him back was influenced by your stutter in way. You should probably find another line of work. I have a stutter and I’m a lawyer. It’s ok to be scared to speak but it’s not ok to not speak because you are scared.