If you use a lot of filler words he 100% doesn't know that you stutter. He may have been poking fun like "Oh it's one of those basic people that say ummmm a lot" Just tell him