I know how you feel. I have the same problem for other things and I think it mostly extends from self-consciousness and low self-esteem so that I care too much about what others think in an effort to make myself more "worthy" or more acceptable. I have learned that other people aren't as aware of me as I am of myself. I know that thought seems silly to some people, but for years I felt like other people were as aware of me as I was of myself but they really aren't. A lot of this thought process was reinforced by childhood experiences and rejection and just plain standing out because of my quietness in certain places and events. If you want candy, get candy and get the ones you like. Those other people can think what they want; they don't know how much you eat of it, they don't know when, and it isn't their business. I am ashamed that I have had thoughts about the contents of others' shopping carts especially last year when I was obsessed with the faults of the food industry and was obsessed with what I fed myself and my family, but those people were probably looking at my cart too thinking I was one of those snobby health-nut types and for a while I was "guilty" of that. And I was wrong for that. We think things about ourselves and about others and others think things about us and it's just the way it is. I am learning to no longer be so worried about what others think; my problems with that mostly extend from the way I was raised. My mom was nearly paranoid about it and my father probably has Asperger's so you can imagine the combination that made and my own social mistakes and difficulties created the perfect storm of worry.
Worry about the opinions of people who you know care about you as a person, a real person with positives and negatives, and give credence only to how those people think of you. I know it's hard. For some reason, I tend to care more about a stranger's opinion than that of those closest to me and even feel more attracted to strangers often than to those I actually have a relationship with. Maybe that's because so much hinges on "first impressions" and it is so much harder for me to make a positive first impression especially when I was younger. Maybe you share this experience.
Over all, I was startlingly resistant to peer-pressure when I was younger. But it is the oddest things I get self-conscious of. and other times it's because I learned to be ashamed or self-conscious of something because of ridicule or rejection or strong reactions from others that sort of scared me or something.