No, it is not at all related to brain damage and retardation. From what I have observed, stutterers are more intelligent than the average population. I myself have an IQ of 136 and have been a stutterer since the age of 4. Further, maths and science have been my favorite subjects.