Is the thing bothering you that you appear to be his second choice? It’s rare to find someone you connect with so I’d either get closure by telling him how you feel and how he hurt you, or just forget about it and see if this goes anywhere.