Are you sure it is because of your stutter? I have joined a close knit team before, they had been together for years, had lots of shared stories etc. It took me a while before I was trusted and became one the team.