I think it's tough for us as ND to separate what could be a career interest or just a passing fascination (it is for me anyway). I'm currently in my 3rd year of Astrophysics, last year I had the same worries as you. I was doubting whether all the stress and stuff was worth it but after talking to a bunch of my lecturers and reading around on the other stuff I was interested in, I came back to Physics.
My advice: try and actually learn about the stuff your thinking of changing too, by whatever method you choose. If it bores you, you know it isn't right. Also think about how interested you've been over time. For example I knew I'd been fascinated with Physics since I was 14 in high school, so it was likely it was just a temporary freak out.
Whatever happens you can't really go wrong, I always tell people who doubt their degrees to just follow whatever their interested in.
Good luck and hope it all works out.
_________________
"I don't know half of you half as well as I should like, and I like less than half of you half as well as you deserve." - Bilbo Baggins
Probably off challenging as many social norms in as little time as possible