Yes.
My main, lifelong special interest does interfere with my life but... Never in a destructive way. And it almost never got out of hand thanks to limited resources.
My special interest is Crafting... Too conventional to be perceived as weird, but unique enough to be uncommon. When it's too repetitive, it was to be 'admired for diligence' and repetitive enough to become my discrete form of stim. And said stim is 'productive', enough to sell it's finished product.
I don't know, my case is one of those rare positive effect special interest. Crafting made me popular at school, got me connected to other people, gave me relief AND profit, etc.
The times my special interest interfered negatively with my life would be uncommon -- because I almost never talked about it.
But yes, I do have issues with too much focus whether it involves any special interest, obsession, or not.