Letting go of fear doesn't mean to remove stutter fear (or anxiety). Letting go means that the fear is still in your mind and you really experience its discomfort, but you don't care, it doesn't bother you, it's not important in your mind, and the discomfort doesn't have a meaning for you. If you think "I will stutter now" then learn that this fear is not your true intention, real, true or fearful in your mind.