Could it be the way that Americans speak?
I'm not being mean here, because obviously people can't help the way they speak, but I find it difficult to listen to anything voiced by an American.
This might have something to do with my hearing impairment - i don't know - but the American accent sounds quite grating and often over-exaggerated, which I find annoying.
This is why I've hardly seen any American TV shows and watch BBC and ABC exclusively for the British and Australian content.
Have you tried listening to speakers with different accents?
I find I can tolerate a nice British accent very well.
If I had a British accent, I'd probably never shut up

I don't always get irritated by the way Americans talk. I only get that problem when Americans are trying to be entertaining.