Um.. I don't think anyone can tell you who to miss.
When my aunt died a few years back.. (I wonder to this day if she had AS) I didn't really know her incredibly well. I loved her, she had parakeets and she always let me come to her house to play with them. Eventually she gave them to me.
One day my mom had picked up a kitten, very small. Little did I know that animals that small are very susceptible to diseases. It started for a day, it barely moved from the bed we had made for it. Its breathing seemed to get slower each time. I was frantically looking up things to do for the kitten to improve its health, nothing worked and I got to see the first pet I was ever really responsible for die.
I cried ridiculously, when this animal died. When my aunt died, I didn't cry as much. It felt harsh that I could cry over the cat more. It seemed more like it was her time, but to be honest she died fairly young (mid 50's). I loved my aunt more.
I think no one can tell you who to love more.
But even if your grandpa wasn't your friend, if he was nice and raised you, loved you..etc, it is something to consider.