My NT sister and I can totally relate to this situation. Whenever our bigoted uncles visit without calling ahead of time, they enjoy wanting to know everything we've done since their previous visit. For example, when I am trying to some homework for a class, I lock my door in order to not be bothered, but they always win.
Example: "Hey you in there? You doing some homework or something?"
During one visit, they took me to a Burgerville and demanded if my NT sister found herself a man to be with. I know I have written this many times before, but our bigoted uncles believe that the spectrum isn't real, that the only way for a man to truly "man up" (spectrum and NT alike) is to enlist in the US Armed Forces immediately after graduating from high school, and that the only way for a woman to truly be happy is to settle down before doing anything else in her life.
As for you OP, you are absolutely not being a jerk, you just want people keep their distance from you.
_________________
Silly NTs, I have Aspergers, and having Aspergers is gr-r-reat!