My Granddad is particularly bad at that. He will start talking to me, then when I start to stutter; he just starts talking to someone else. You're right, in that they probably don't mean anything bad by it. They just feel bad for you. My Grandparents are the same. Just smile, and carry on the conversation. For me at least, when I get going my stutter tends to lessen.