Personally I think it implies that a stutter makes your words mean less. So I’d say yes it can be offensive.