Yep, I get that too. My brother stays mad at me all the time on the premise of "Hey, you started it. You were already pissed off." When I try to tell him I wasn't at all, he then thinks I'm arguing with him.
And my mother and I tend to get into it a lot for similar reasons. I've figured out that my "astonishment voice" is very like my angry voice. When my mother presents me -- by either something she's said or something she's done -- with an "Are you sh*tting me?!" moment, my reaction and tone of voice will often anger her. She thinks I'm being a smartass or mocking her in some way, when in reality I'm just shocked and amazed by whatever it is that she just said.
_________________
AS: 136/200
NT: 66/200
EQ: 45/50
Go as far as you can see. When you get there, you will see farther.