I can kind of understand how you feel. While I don't stutter, I have social anxiety which I believe shares an overlap with some emotional difficulties that PWS experience. (Not saying everybody with either of these disorders feels the same way). I think that perhaps it's from this idea that we identify as having some long-term "issue" (It's part of how we see ourselves) and when we observe and present ourselves as not having this "issue" we feel like we're not ourselves. It's a twisted psychological idea that our disorder defines who we are as a person, which should not be the case.