I have four kids (one with AS), and yes, it is very, very difficult to not lose your mind among all the chaos. When things get too much for me, especially if they have friends over, I go hide in my office for a while. Funny, but I feel that despite the stress of raising kids, having them has saved me from myself...I've learned invaluable coping skills because I HAD to.
The "empty space" you speak of can be a bit disconcerting -- the rare times my in-laws take the kids overnight, I feel relief and happiness at first, but soon feel restless and anxious. I miss them when they're gone for more than one day. The little dorkheads. 