Actually, there is not enough information with just that phrase alone to take anything out of it. There is a wild array of meanings this could take on (and no, not all of them are negative) depending on the situation that triggered the comment in the first place. For example, I could be constantly teasing someone playfully, and they could playfully tease back saying "You're a horrible little man." In this manner, the "horrible" means that I'm terrible for teasing (and would be such if said person was a stranger), BUT because I know the person and they know I'm only joking (as long as I don't go too far), it more along the lines means "I should be annoyed, but because I know you I'm not." The "little" in this context is then a reference to maturity level, as excessive joking can be seen as a sign of immaturity, a trait of someone younger, hence the word.
But that's just an example, and unless we have the full story, anything we say it means is pure speculation.
_________________
Your neurodiverse (Aspie) score: 91 of 200
Your neurotypical (non-autistic) score: 107 of 200
You seem to have both neurodiverse and neurotypical traits