Observe non-stutterers for a week and you will see that many of them hesitate and/or repeat words/ syllables all the time, without making a big problem out of it. Don't be so hard on yourself.