I don't have a problem remembering names (but I do have a problem with faces).
Due to hating my name I always had problems introducing myself and react to being called upon. I dreaded giving anyone my name because then they'd start using it, and I honest to god found anything a bully could throw at me less offensive. My original given name is the ugliest ever IMO. But since having my name legally changed, I haven't had that problem.
With the names of others, I do tend to use 'hey' or 'you' more than their name. I use people's names very little wen talking to them. I'm not sure why. It's even more pronounced if I dislike the name, not to mention if it's my original given name, or any nickname of it.
I'd rather under-do it than overdo it though. I know someone who uses people's names in every other sentence she utters. That's really annoying.
If there are only two people present, I do find it a bit superfluous to use their name. After all, who else would you be talking to?