I'll echo the others in thinking it's just general shyness and not probably about you as being "odd." It seems to me that very small children might notice differences among us without judging them the way older kids do. Even really obvious physical disabilities, the really little ones just seem to take in their stride.
Take me with a grain of salt though.... I have an opinion on everything ... I am not a mother or a teacher, I'm the youngest child in my family, I don't have a lot of experience with children.
I am told that when I was four years old, sitting at the dinner table with my (white) family, sitting next to a guest who was a close friend of the family, i.e., I had met him very many times already, I looked up into his (black) face and said "Jim! Your hand is black!" I had never noticed until that minute........