I usually have to meet someone half a dozen times before I start to recognize them on sight, and then I can still be thrown again if they change their hair, or wear radically different clothing, or I see them out of context - somewhere I would not expect to see them. I might recognize a colleague in the office, but if I saw them walking down the street, I could still be unsure as to whether it was definitely them. I'm sure that people have thought I was being rude sometimes because I seemed 'off' or distant when they approached and started talking to me, but, in truth, I had no idea who they were, although they clearly knew me.
80% of people look very much the same to me.