It's an insult that is often used, but it is not to be taken literally. Many people say this when they think someone did something silly, even when they know the person they are speaking to isn't actually intellectually impaired. It's not a nice thing to say, for sure, but he was not implying that you actually have an intellectual impairment. And no, you can not have an intellectual impairment with an Aspergers diagnosis, you must have average or above average intelligence to have Aspergers.