When you're being "fluent", are you achieving fluency by world substitution or something? If you're still trying to control your stutter in fluent periods, maybe speech comes off as odd? Or maybe you have rude friends. I'm quite severe and I know this so Im not a blabber mouth but when I talk they all patiently listen.