The thing is having a stutter you have to learn to be firm more than anybody else in order for people to take you seriously. When you were with your friend and you agreed and didn't want to cause conflict, that was the worst thing you could have done. If you said no you're creating a conflict that is beneficial to both you and your friend as you can both learn to become better with each other. Before you deal with anything else, you have to learn to be honest with your stutter, as at the end of the day (as I've come to learn), the people you interact with are merely a reflection of yourself and how you molded them to act towards you. If you don't understand I can elaborate more on what I mean