That would be a different answer for different people. It would depend more on the local culture than the size of the city. I've lived in several different cities, and I'm currently in New York. New York seems to be the best fit for my personality. My experience with New York is that if you don't bother anybody, they won't bother you. New Yorkers have a reputation for being rude, but I've never encountered a person in New York that was rude to me just for the sake of being rude.
Other cities I've lived in or near were Denver, LA, Daytona Beach, and Jacksonville, FL. I didn't really like LA, Denver was OK, I somewhat liked Daytona Beach, but Jacksonville, FL is a hellhole. I'll say this about Jacksonville, if you don't like the way guests behave on the "Jerry Springer Show", Jacksonville isn't your town. When I was there, there was always somebody getting in my face.
Move around until you find a place you like.