What your colleagues feel isn't your fault, nor is it your responsibility to shield them/make them feel better. If they feel embarrassed or nervous, that's their problem. Try not to burden yourself with that kind of stuff. It isn't worth your time, especially when you have your own experiences to deal with. I know it is easier said than done, but for real, their feelings about your stutter don't matter.