I don't like math, but based on what you describe you are not addicted, just an enthusiastic lover of it. It's a hobby or interest of yours. Your therapist obviously really hates math, probably never hung out much with math whizzes, and because of that, believes that any one who likes math has an unhealthy obsession with it. This is a kind of tunnel vision on the therapist's part. You can either accept that you and the therapist will never agree about your interest in math, and continue with him/her, or get another therapist.
It does sound like you will interact with people when you need or want to, but that your desire to is somewhat limited. That is common to those on the spectrum, including myself. I am a hermit type of person. I consider myself non social, but not anti social. However, I will interact with people when I need or want to, but like you, I don't want to that much. I wave at and talk to the neighbors when I occasionally go outside, and talk to people in the stores when I occasionally run errands, and talk to relatives when they call or come to visit, but I will never be a social butterfly. Not every one is an extrovert. Unfortunately, my father is, and labors under the mis-belief that all introverts are miserable because they are introverts, so he keeps trying to get me to become an extrovert, so I can be happy. I am in my mid 50s, and am more likely to be struck by lightning, picked up by a tornado, swept over a water fall in a flooded out, burning building, than I am to become an extrovert at my age. My being alone does not make me unhappy. I was very unhappy when I had to live with relatives. I felt much better once I was finally able to live alone. Granted, being an introvert is not what society considers the "norm", but if it works okay for you, and doesn't hurt other people, then there is nothing wrong with your being an introvert.
Although I hate math, I do recognize it's value in so many areas. We need it to pay for any thing we buy. Math got us to the Moon, and back again. It is used by surveyors, bookkeepers, and in most other fields, even in cooking. Yes, you need math in cooking, when measuring ingredients, and cooking times, and if you want to change the size of the batch, you need to recalculate how much you will need of every ingredient. So, while you don't have a math addiction, it looks like your therapist has a math phobia. 