It sounds like he is discriminating against you based off of a disability or you feel that he is going to hold you back based off of that discrimination. That is not ok and, depending on the job, a punishable offense. I am in no position to tell you what to do, but if you feel like you are being discriminated for your stutter, HR needs to know.