The difference isn’t really surprising. If humans weren’t so inquisitive and hadn’t developed the ability of language, a penis would hardly receive visual attention at all. Breasts, on the other hand, have long been a very strong cue for males, in addition to their primary use for lactating. Unless there’s any exception I’m not aware of, women are the only female mammals with permanently bulging mammary glands. It’s been hypothesized they evolved this way because they mimic the shape of the buttocks, which males already found attractive before our ancestors became bipedal, as it was what they saw when approaching a potential mate from behind.
There’s no doubt breasts have undergone a lot of sexual selection, and, in turn, males who tended to find them relatively dull have been outbred by those addicted to them. In other words, evolution has designed them to be beautiful, and us to find them so; the latter probably includes women themselves to some extent, as a side effect.
I think it actually goes beyond mere sexual stimulation. As a little boy, there was a time when I thought the only body parts which always had to be covered in public were those involved in excretion. When my mother told me adult women had to cover their breasts, too, I was quite surprised, because there didn’t seem to be anything “dirty” about them. It became even more puzzling the day I found, in my grandparents’ house, a perfume vial with a mermaid figurine as a cap. The mermaid was fully naked and I spent a few minutes in awe, thinking just how beautiful her breasts were and how strange it was that they had to be hidden. Needless to say, I had no concept of sex yet, nor would I have it for a good handful of years.