Most of the time I wear whatever I want that's comfortable. I am no longer able to work because of other health issues, and rarely go out, so this isn't a problem around the house. When going out, I wear a nice pair of slacks, and a nice top. If it's cool or cold, I add a sweater or coat. If my hair is in a long, shaggy state, I wear a head band when going out, and a hat if it's cold. When going out, I pick slacks and tops that go together, color wise. Don't have dozens of shoes in different colors, as that never made sense to me. Have a pair of mostly white running shoes, a black pair of running shoes that can pass as shoes, a pair of sandals, and a couple pair of slippers. I've never been one to really dress up. The last time I dressed fancy was several decades ago, for my eighth grade graduation, and I only wore that dress that one time. What a waste of money. There were a couple of times after that, when I wore a navy skirt. Once was for the eighth grade graduation class party, and a year or two later I wore the same skirt for a relative's wedding. It wasn't a real big affair, as both bride and groom were elderly and widowed. Old as they were, they managed to live a good many years more. She was my favorite great aunt, so I was willing to wear the skirt and attend a social function because it was for her. I even remember her first husband, who died decades ago. I liked him more than her second husband, but I have nothing against her second husband. It has to do with the type of people they were.
Where am I? AAAAAHHHHH!! !! ! I've entered the digression zone! Sorry. 
Basically, I mostly ignore fashion, and wear what I like that is also comfy. I also don't like skirts and dresses, and don't look good in them anyway. People don't always like the way I dress, but we each get to choose what we wear, and I do dress somewhat better when going out. I am under no obligation to wear what others want me to wear. The one exception would be job related, and I always dressed within job requirements. Of course, I never applied for jobs at any place fancy, as I hate dressing fancy, and can't afford a whole new wardrobe of fancy clothes anyway.
My advice is to wear what you want, within reasonable limits. There is plenty of room within "reasonable" limits.
_________________
If a man does not keep pace with his companions, perhaps it is because he hears a different drummer.
Let him step to the music which he hears, however measured, or far away.--Henry David Thoreau