That sounds like me. I don't really have the anxiety when speaking to my mother but I still stutter. I think after a decade of doing it it's become a habit or possibly something to do with strengthening the pathways in the brain. I've seen quite a few stutterers say they are confident including myself but maybe it's not social anxiety they have and it's actually a more specific form in speaking anxiety