I think at some point you have to be direct with the guy and it's more merciful to do it sooner rather than later. It's not being kind to hang out with someone out of pity or guilt. You are just delaying the inevitable and the longer this goes on, the worse it will be. He is not going to take a hint. Just let him know that you don't want to hang out anymore. And if he asks why explain that you don't have much in common and you prefer to hang out with people who share your own interests.
The real problem is that your mother pushed you into this. If you don't address that somehow it will most likely happen again, maybe in different circumstances.