In all seriousness this phrase, as it relates to the majority of the fluent population, makes sense. You say that it is used in a way that implies disfluency = unintelligent, but I think it has more to do with nervousness or indecisiveness since that is when most fluent people tend to stutter. Of course getting stuck with those types of labels due to the way we talk isn't great either.