I do this too, sometimes they even let out a little laugh. It feels better when you can speak without them not understanding why it took you 10 seconds to say "no thanks." Less awkward.