It's very likely not the same that's happening when a non-stutterer listens to a stutterer, so don't worry, there's very likely nothing hypocritical about it (and if so, maybe only in relation to a very small subset of instances).