Perhaps it's a subconscious association between height and dominance, dominance and _______ (something sexual/romantic).
Something I have definitely noticed with a lot of women is, it doesn't matter if you're short fat and toothless with green skin and a perm, if you can make them laugh and keep them engaged and interested you will get dates.
It seems to be harder for people the further they are from the typical idea of the ideal mate of the people they are trying to attract... but it's not as much of a factor as people think.
I have dreadlocks. Women don't tend to like them, they mistakenly think dreadlocks are dirty or that they make me a hippy, that I do lots of drugs or that I am stupid because of them... so I know how it feels to not generally be women's ideal match, but look at it as a filter if you can! The ones who won't want to be with you because of your height are people you won't want to be with, maybe it's because they're shallow or judgmental, maybe it's just because they aren't attracted to you because of it or something else (so you two are not a good match...) don't focus on why you aren't a good match for them, you're just not! Accept it, it's fine!
I think most men are attracted to larger breasts because it indicates to their subconscious that this woman is able to provide for his possible future children's nutritive needs... it's an instinct left over from when it was relevant to survival. Maybe women's general subconscious liking of tall men is similar and to do with a tall man's mechanical advantage in terms of leverage (which is relevant to all sorts of things including hunting and fighting) or their ability to forage (gathering fruit nuts of berries a shorter man could not reach) or otherwise to do with your ability to protect them and/or your children?
Sorry for the comments in the middle if you were asking this in a more general/philosophical way and not a personal one as it seems to be... I tried to cover both angles.