Yes I relate so much to this. I do this as well and it makes them feel uncomfortable more so than me and helps a lot with fluency( I know this may sound a bit mean,using their discomfort to help myself but hey,if it works it works). It really does work and has helped me a lot in social situations where there was a lot of attention on my stutter