You don't need to apologize for anything. He is mostly at fault here...what he did to you was kind of rude.
It appears that during the time you've described, he was only focusing on what HE wanted to do and did not take your opinions and preferences into account. Have you told him that you get tired after a while of socializing and don't like being out for long periods of time? If so, you absolutely need to tell him this...otherwise he will keep doing this to you more.
If he is the kind of person who loves being around people and stimulating environments all of the time, he may not be the best person to be with long-term. I'm sorry if that's a little blunt, but part of being in a couple involves sharing and preferring similar activities together. However, this can be changed, provided that you both have an open and honest discussion about activities you can do together. He needs to be willing to work with your needs and to compromise if your interests and preferences clash. He needs to be willing to accommodate you...other wise he is just getting his way. Relationships are a two-way street and each person needs to be respected and to feel valued by the other person. It doesn't seem like he is valuing you here because he is doing only what he wants and only seems to be coming back to you for sex. That doesn't sound healthy.
_________________
Given a “tentative” diagnosis as a child as I needed services at school for what was later correctly discovered to be a major anxiety disorder.
This misdiagnosis caused me significant stress, which lessened upon finding out the truth about myself from my current and past long-term therapists - that I am an anxious and highly sensitive person but do not have an autism spectrum disorder.
My diagnoses - social anxiety disorder and obsessive-compulsive disorder.
I’m no longer involved with the ASD world.