I can relate to that, Quantum. As a child I was interested in (but not a special interest and not obsessed with) astronomy. In junior high a classmate knew more about it than me, and that made me lose interest in it.
For the last 22 years or so, turtles have been my special interest. I have never met anyone IRL who loves turtles, so IDK how I'd react if I did, but online I follow turtle sites and it doesn't make me feel anything negative (other than when it comes to how bad the situation is for the endangered ones and how people's dumbness hurt turtles, but that's different). So maybe that wouldn't be the same IRL either, but I can't say. I might just resent them a bit for doing what I can't do and likely never will get the opportunity to do.
I love what they're doing for the turtles, so I hope I wouldn't.
I can definitely be jealous of people, and not restricted to relationships.