If I do look at people's faces I either know them extremely well or I am forcing myself to.
When I force the issue on myself, it makes me think of other things (like their facial features) and I won't be able to listen to what they're saying.
I believe that my inability to look at people gives me face-blindness which means I don't remember them and this, then, creates other issues.
I find it very confronting when people get in too close to me, when speaking, and I always take a step back or turn myself sideways so that they are literally speaking into my ear.
I also have a keen sense of smell and will usually concentrate on people's smells which is very distracting in conversation. Especially so if their odour repulses me and all I'm thinking is: "I have got to get away!"
At least, sometimes, I can remember people by how they smell.

you could be speaking for ME! i notice body smells too, and some people i don't like to hug, like if i don't know them that well, because of their smell. not body odour, just their own smells.
i often turn away completely as well, especially if it is an important conversation or an emotionally charged one.