Of course it's possible! I lived in Japan for one year and it was sooooo hard. I stammer a lot on words beginning with vowels which is like 60% of the Japanese language so fuck my life basically. But I just did it! Now my language skills are better than they have ever been! Believe in yourself honestly! I was like you. I thought 'why tf would I go to uni to learn Japanese when I can't even speak English (my first language) fluently?' But you've just gotta go for it! Otherwise you'll spend your life thinking about why you didn't go for it.