Yes, I have been told "you're a fag" several times in my youth - the last time I recall hearing it was at the age of 23, and it started when I was 14. One guy at a house party at college, who'd been harassing me on and off about it, sarcastically yelled at me that he wanted to "do me up the a**" and got his imbecile buddies to laugh along, I guess he wanted an audience, but it sucked for me.
At the age of 18, another guy in a grocery store where I worked part-time would yell to my co-workers that I was a "limp-wrist" - it made no sense to me at the time, I thought mabye he's just a latent homosexual (which is true in many cases, the closet-case becomes the persecutor), but in retrospect I surmised that it had something to do with my nonverbal & motor-skill deficiencies being observed while I worked.
Remember what Seinfeld said: "guys who are thin, neat, and single are thought to be gay". Well, I was perpetually single, but well built, and not really neat - I was kind of messy like your absent-minded professor. I was also a moderate follower of sports, and enjoyed cars, and went to the gym fairly regularly. So, process of elimination, it clearly came down to my non-verbal and motor skill deficiencies.