I'm kinda similar. I'm pretty sure the majority of us that stutter have some sort of social anxiety disorder. Whether it causes the stutter or the stutter causes it is another topic, but anxiety definitely affects our fluency.  My job involves lots of phone calls to people in the health care field. I'm super comfortable with all the topics I am discussing with anyone, so that part of the conversation typically contains very minimal speech disfluency on my part. The wildcard part for me is the first 10-15 second introduction of the conversation. If I speaking with a physician then I'll usually stutter a bit. With a PA/NP it's minimal. With a nurse or patient it's almost non-existent, but will happen rarely.