I'm fine.
At a previous job, people used to frequently go out to eat as a small group and I nearly always felt left out when not invited along. Obviously, it is hard to fit in at work if you aren't accepted by the others.
Now, twenty years later, I don't care if people do go out and eat. I'm going to do what I want to do instead. Most of the time at this workplace, if we have a group meal, someone sends out for pizza or cooks something in the office kitchen.