If this were an "Am I the asshole" the consensus would be yes. You like this person but their stutter inconveniences you and you don't like that. Nevermind how difficult having a severe stutter must be to the other person, time is money and you feel drained by them (a fear I'm sure they already have). You're honest, but I don't think you're able to be a good friend to this person (by your own admission). It's a good thing to recognize in yourself and I do appreciate you coming here and asking for advice, but I don't think there's anything to say other than to be patient with people and be more interested in what they're saying than how long it takes. For the record I don't think you're an asshole as a person, but your feelings about his stutter and how it affects you comes off as selfish and unkind