i get something similar, which is people (co-workers, usually) expressing hostility because i zone out and i'm not interacting with them constantly. i do it because i have to, to avoid the steady building frustration about not knowing what their interactions with each other are all about, etc. maybe they're joking, but people accuse me of hating them and that kind of thing all the time. which makes me angry, which makes me withdraw more.
i can't win. and no one makes any effort to understand - they just accuse.
_________________
Now a penguin may look very strange in a living room, but a living room looks very strange to a penguin.