Should we break up?
I want to offer a more positive outlook on things. No matter who the people involved are, and whether they have ASD, something else, or not, there are always differences in preference between two partners.
Compromise MUST exist in any relationship.
Maybe you guys can have a conversation about compromising.
Maybe he can take a day per week to go hang out with friends, and you guys can designate a few days per week just to hang out by yourselves. Maybe 2/3 or 3/4 of the time you guys go out it can be alone, and 1/3 or 1/4 of the time you can invite along some friends.
If going to an exercise class doesn't interest you, maybe you can find a hobby that you enjoy to do while he is there. Maybe it can be your time to do something that you like which he doesn't enjoy. It doesn't even have to be at the same time as his exercise class if the timing doesn't work. Your compromise can be that each of you take a little time each day to pursue a hobby.
These are all just ideas. The point is that you both should take turns engaging with each other in ways that primarily benefits the other person. I would caution you away from trying to find someone perfect for you, because no one will ever be 100% in sync with you all of the time.
It's only definitely over if no compromise can be made, or one or both people are unwilling to make any sacrifices.