in a single term: information overload.
To me, eye contact is an extremely intimate experience, almost on the level of cuddling with someone, and certainly not something I can easily pull off with a stranger... it just feels... wrong. The amount of information contained in an individual's eyes, (or at least the amount I seem to perceive) can easily be overwhelming, and I sometimes fear that others can "read" mine like a book. It has never been a natural thing for me to make eye contact, and so it requires a conscious effort most of the time, which of course takes considerable cognitive resources, which in turn decreases my ability to follow a conversation or do much of anything else.
Some people say that the eyes are the window to the soul, and I can easily see how that could be seen as true. When looking into someone's eyes, (in the case of someone I am familiar and comfortable with), I can get entirely lost in them. The details and subtle patterns in an individual's eyes can easily leave me entirely mesmerized for extended periods of time, and the experience can flat out remove my ability to think.
If it is someone I am unfamiliar with, even brief eye contact produces an awkward, almost electric feeling. I can see how some would describe it as painful - it is almost like someone electrocuting the front of my brain/backs of my eyes. I still notice the patterns and details, but it feels highly inappropriate to me. Add in to this the fact that I still get absorbed into the patterns and the fact that it requires a large amount of effort to actually maintain eye contact and I am again left cognitively incapacitated.
_________________
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind."
-Dr. Seuss