So this was on my mind because my sister brought it up more generally (not the part I'm going to describe, but the houses we lived in as children and teenagers).
I lived in one house from about 1971 until 1980, and then another from 1980 until 1985. These two houses are pretty where all my childhood memories are, good and bad.
Several years ago I decided to take a look at them. I mean, I was living less than a mile from the 80s house (and my old middle school was behind my backyard, so that's how close) and two of my close friends lived very close to the 70s house.
The 70s house, when I lived in it was a kind of yellowish tan color, had a porch that extended from the front door, around to the right side (facing the house from the street), with a picture window facing the street and a second picture window facing the driveway to the right. The driveway itself had gravel rather than paving. The lawn was grass with some small flowering trees planted around the porch. It had two stories and an attic, with a bedroom window looking out onto the street on the second floor, and attic windows above that. There was also a sloping roof marking each floor, right below the windows - I actually climbed out of the bedroom windows and onto the roof, and got into some trouble. I had some exciting daredevil instincts before I got older. Anyway, along the left side of the house was a narrow dirt path leading to the back yard.
When I went back to look at this, the lawn had been landscaped with stepping stones and barkdust, bushes, and flowers. The trees next to the porch were gone. The house itself had been painted a peach-ish color, and the path along the left had been filled with trees. The driveway had been paved over. At the time I saw this? I couldn't actually tell whether I was at the right house or not. I came to the conclusion that it had to be the correct house because it was too old to be recent construction, and the architecture matched the house I remembered. Now when I look at it on google street view, it's recognizable to me because I know all the points of similarity to look for, but the first time was a shock.
The 80s house was a two-story house with a balcony with two gigantic yards - one on each side, big enough to hold another entire house. The house it self was a kind of bluish-gray, and had a fairly tall evergreen in the yard to the right, toward the sidewalk, and a mature apple tree in the yard to the left, toward the back part of the yard. When I went to look at this house, there was a house with cyclone fence to the right, facing the cross street, and a landscaped garden with an iron fence to the left. There had been a church on the far side of the yard to the left, and it is now a house. The house itself is still there, the paint is the same - and peeling. There's a driveway to the left of the house. I still have trouble recognizing this house, I have to check over everything a couple times to make sure it's right because the surrounding context is so different.
... Similarly, whenever someone rearranges the furniture, I have to pause and work out what's wrong because I do not quite recognize what I see.
I don't usually get upset, but I get visibly confused, at least enough for someone to outright tell me, "We rearranged the furniture."
On the other hand, when I look for something that should be in its usual place (dishes, food in the refrigerator) and it's not where it should be my brain starts to take off in a kind of frustrated escalation from which I have to consciously talk myself down. I once pissed off my niece badly because she thought I had told her to calm down when I was about to lose my temper because the paper plates were not where I expected them to be.
My parents used to clean my room a lot when I child, rearrange it, swap my sister and I from one bedroom to another. This was pretty stressful. I suspect they didn't want me to find anything, ever.