That's because to be called "colored" in the old days in the US meant that you were "black," basically. It was used as an insult, as well as a "descriptor."
It was almost the same as being called the "N"-word at times. At other times, it was the same as being called a "negro."
"Person of color" has quite a different connotation. It has never been an insult---only a descriptor.