Unfamiliar territory makes me a bit nervous, as do situations in which I'm expected to be social. It's been many years since I've gone to someone's home for the first time, so I'm not even sure how I'd react. I don't get out much, and definitely prefer it that way.
One reason I don't like other people's homes is sensory. I have to deal with colors, sounds, smells, and such that I would prefer never to experience. Walls painted red, clocks ticking away or lights buzzing above, and odors of all sorts. Combine that with people who know each other, and I get very quiet very quickly, and very tired very quickly.
I was just talking to a friend about how I liked his place because he has made virtually no changes to it in the past 10 years. I told him that I appreciate the constancy of it, the stability and predictability of it. My mother's home, by contrast, changes by the month, which I find disconcerting.
Finally, food. Most people serve food when you visit their homes, and most of what is served does not fit my idea of food, nor is it served in the time or manner that I eat. So I have to somehow politely bow out of actually having anything more than a glass or water or cup of tea.
At least I'm fortunate in that my family doesn't do Christmas.