tl;dr: because sexism.
We can crudely divide people in two groups: the conservatives, who see women as private property (the property of her father, and then, her husband); and the liberals, who see women as public property (the property of any man who wants her). There are, of course, the people who don't see women as anyone's property, because women are people.
Conservatism has made sure women can't expose too much of themselves publicly (remember, they are one man's private property, other men have no right to see her!). This kind of thinking used to be stronger in the past, and still is in some places, even taken to the extreme in countries with sharia law, for example.
However, liberalism has been rising in the west, and so women have been exposing more and more of their body publicly. This can be nice for women (wearing fresher clothes when it's warm) and also not nice (men thinking that they can do whatever they want to women who expose their bodies, since they're public property).
Please note how women's clothes tend to be much more revealing than men's - shorter shorts, shorter sleeves on t-shirts, see-through clothes, wide/long necklines, tight clothing in general, clothes with random pieces missing, even smaller and tighter underwear... very common in the women's section, almost non-existent in the men's section. This is due to the advance of liberalism; women are public property now, so they should expose their bodies for all men to see and do as they see fit.
But if women are supposed to show off their bodies so that men can feel good by looking at them/use it as an excuse to do whatever they want to them, then why can't we be shirtless in public?!
We can also think about not wearing bras. By some, it's seen as titillating. By some, it's seen as gross. It may seem like those are conflicting opinions, but I think it's kind of obvious: perky, full breasts are considered sexy and good-looking, while... other kinds of breasts (droopy, uneven, hairy, too far apart, too small, too big, with stretch marks, dark nipples, etc) are considered ugly, gross and unacceptable. Men want to see sexy breasts - liberal men on the streets, conservative men at home. But they don't want to have to see ugly breasts, whether it's by being braless or shirtless. There is much division on that, in the same way. Think about how fat women get shamed and ridiculed for wearing some of today's trendy clothes (crop tops, short shorts/skirts/dresses, tight clothing, etc): these women aren't considered good-looking, so they're not supposed to be seen. Men want to see sexy, not ugly. This is also why women breastfeeding in public is a controversial topic: if the breasts aren't sexy (breastfeeding is seen as not-sexy, or at least inappropriately sexy by most men), men don't want to see them.
If women are supposed to be men's property and servants, we can't be free to wear whatever we want. And men can't, either, lest they be associated with womanhood (men performing femininity are ridiculed, because it's seen as a superior being emulating an inferior being - not acceptable). Those rules are needed to maintain patriarchy.
Sorry for writing such a long post! ^-^ I hope to have added to the discussion.