I don't have much faith in IQ tests. The whole idea of something as complex as the human mind having some fixed horsepower that can be reduced to a "stupid number" (as you neatly put it) seems absurd to me, and I would question the motives of people who push the view that IQ tests are a good thing and want everybody to get measured for it. The only use I can see for it is as a quick and dirty method for assessing some kinds of brain damage, and I think the changes would need to be pretty huge before I'd take the result seriously on its own. I don't know what my IQ is and I don't especially want to know. I'm more comfortable with aptitude tests because they define the kind of task and might give useful insights into which horses are a good match for which courses.
I think stress, depression, and numerous other mental states can markedly affect performance in a lot of mental tests, and that many tests are especially unsuitable for people with ASD, because as you suspect, those tests were never designed for us. I got strange results for a "working memory" test where I scored badly on my first attempt, fair on my second, and excellent on my third. And that test was supposed to distill out a single, absolute score that shouldn't have bounced around as a result of factors the test designer neither knew about nor made allowances for. I think what happened was that I somehow limbered up and got used to the remit as I proceeded, that my mind adapted gradually to the challenge.
I don't know specifically why your background experiences affected your IQ test the way it did. Superficially it looks rather paradoxical, but I don't think the mind responds all that simply to circumstances. One person can be weakened by a particular challenge while another can be galvanised by it, then as time goes by you might see the weakened one overtake the galvanised one. Plus of course, the entire IQ concept seems dodgy in the first place.