Well, Aspies like me--the super-cerebral "little professor" types--usually don't irritate me. I might get a little bored if they talked endlessly about something I am not interested in, but I wouldn't get irritated unless they were a jerk to me or they say something that is factual wrong.
But other people on the spectrum--HFA, Aspie, whatever they want to call themselves--do irritate me often, to be honest. These people in my experience tend to be more emotionally demonstrative than they probably realize. I don't relate well to them at all and I find dealing with them very draining. I also am quite tired of the "I hate how people think all autistics are suppose to be geniuses!! !" crap. I honestly don't care how smart anyone on the spectrum is. I just know I am sick of the whining and projection. It's their insecurity and need for validation, and rather than confront that insecurity, they try to tear down other people on the spectrum. Just goes to show, we're not really that much different from neurotypicals when it comes to psychology and egos!
I'm not very O/C or bipolar. Emotionally, I'm quite stable and non-reactive, despite my low threshold for stress. If I'm not stressed, I'm downright zen-like by nature. Aspies who have more OCD or biploar traits tend to be more easily irritated than me.