You might be in the minority when it comes to that.
I generally do not go into a panic when derealization sets in with me... I tend to have a pretty consistent and profound sense of detach when it comes to my body and brain, so I'm kinda used to it. Still, it's not a good thing to me. It causes significant disruptions in my own ability to consistently be functional. Some days I do fine and plow through life, doing it all on automatic, other days I'm lost and spend all my time trying to ground and center. Therein lies the problem for me. If I cannot employ logic in the haze, I'm screwed and end up doing nothing all day except sit around and go, "Woah" or something. I do not always have someone in my house to help me on my bad days either. That can also cause problems.
For a lot of people, it is really a disturbing thing to have happen, especially if it is a new and sudden thing. I imagine it would be like a sighted person suddenly going blind. If you have always been blind, waking one day to not see is no big deal, you know? You already have knowledge on how to navigate life that way. Plus, it can make people think they are legitimately going crazy when it first happens. That is never a settling way to feel.
So yeah, I think talking about it like it's a bad thing comes from things like that... hard to function and scary.
_________________
"...don't ask me why it's just the nature of my groove..."