It's definitely interesting to think about! My boyfriend sometimes stutters after we've been hanging out for a prolonged period, so I think it's definitely a psychological 'thing' that can sometimes happen to a non-stutterer. I wouldn't worry too much about it though, you seem like a good person who's concerned about not making your neighborhood more uncomfortable. Your neighbour probably appreciates the effort you're making to talking with them:)