People don't hate Rain Man, they know he's a person too... they just hate the way he gets treated and so don't want to seem like him for fear they get treated the same way. At least that's how it is for me.
As for Big Bang Theory, I agree that it's neither the best nor the worst starting point and that certain reading would be better, that BBT is one sided.
But Tony Attwood's stuff is one-sided too.
And teachers and mental health workers who are mentally impaired don't necessarily have an impairment that will effect their job.