I couldn't say thank you

I couldn't say thank you Someone went out of their way to hold the door open for me today and I couldn't say thank you. I managed to finally push it out after 5-10 seconds after the guy was walking away. Before I said it he kind glanced over his shoulder and looked a bit pissed off. I felt like such a dickhead but what can you do? I just needed to vent this on here because it keeps playing over in my head and it's driving me insane because I've got a job in the same place as him now, which is making me paranoid that everytime he passes me he will think I'm rude or ungrateful.