No, it wasn't stupid. I've seen countless stories on this subreddit of people being effective teachers with a stutter. It might take work, but I'm sure there's something you love about teaching that you wouldn't want to miss out on!