I definitely feel the same way.
Having a loud two year old brother, mostly annoying sister, and a down syndrome brother, I dot have much of a downtime schedule...
Which is very frustrating. Sometimes I have to stay up until midnight in order to feel like I have had a good amount of downtime... Though I do get it, just scattered throughout the day.
My family knows which parts of my day they may tinker with, and for the most part, what my limits are. Do a few chores without the parents asking and I am able to program my heart away all day...
13, and home schooled, so I never have to leave home 
_________________
Never leaving,
Aspergers + Synesthesia
IQ 142