So...
You have to understand how primal, and generally unintelligent most people are.
If they don't see you constantly chasing skirts - like it's the only reason to be alive - then they're assuming that you have some alternate sexuality. If they have little or no theory of life beyond chasing skirts and whatever your life experience is that's off of their particular set of cognitive comprehension maps, then they're going to try to reduce and 'thumbnail' whatever it is you actually are into something they can understand, ie. if you aren't out trying to actively bang women all day - in front of them, then you must be sexually 'into' something else.
What you're telling us is that you're not showing a feminine side enough for them to think you're gay, otherwise they probably would have gone for that first. OTOH if you don't seem gay but you also seem quiet and introverted - pedophile is a quick and easy thing to reach for where they can make a mental map of you that's at their intelligence level.
It's depressing stuff but this is really what I'm coming to understand, ie. that if you have any complexity in your life that they can't account for they'll need to 'make up' or 'fabricate' an explanation for your behavior that can be processed from their level of intelligence. None of it needs to be true, it just has to be a 'good enough' model to satisfy their level of interest and the model they choose, how it effects you, is of minimal importance to them. They don't think a lot, they watch a lot of TV, garbage-in garbage-out.
_________________
“Love takes off the masks that we fear we cannot live without and know we cannot live within. I use the word "love" here not merely in the personal sense but as a state of being, or a state of grace - not in the infantile American sense of being made happy but in the tough and universal sense of quest and daring and growth.” - James Baldwin