He was all those things, to different people. There are people who think I'm nice. There are people who think I'm a witch, with a "B". There are people who think I'm stupid. There are people who think I'm smart. There are people who think I'm weird.....
One of my sisters and I have shared stories about things that happened when we were growing-up, together----and, I've often wondered what she was talking about, because her account was so very different from mine. The point is we all see, hear, feel, with the experiences we've had----and, what we've gained / lost, having those experiences.
When someone saw your grand-dad as charming, they (your grand-dad and the person) connected in ways, maybe, that he didn't connect with other people. Also, if someone saw him as a bully, maybe the other person was weak----one can't bully someone, unless the person allows the bullying. The same with the rest of your adjectives..... Your grand-dad felt differently about different people, and therefore, responded differently to different people----and, conversely, different people felt differently about him, and responded differently TO him.