Yeah avoidance behavior is never recommended. What OP is doing is reinforcing their fear of stuttering. The only way to get over a fear is to go through it.