Don't quit. No matter how hard it is. Eventually you will gain confidence when you keep talking to the people. Even if you lose the job after some time, you'd still have gained a lot of confidence for trying.