As naturalplastic says, it's a word that used to be used for anyone who was physically indisposed due to injury or illness, permanently or temporarily. "A person suffering from disablement or chronic ill health."
It used to be a normal, acceptable word a long time ago, merely a descriptive similar in meaning to any term for unwell people or sometimes even wheelchair bound people. People might have said "That area there is the invalid parking" the way they would today say "That area there is for disabled parking."
Our Physical Education teacher used to "joke" about the people with a sick note to dismiss them from gym class, saying "Invalids can read a book over here at the side."
I believe the word passed into being considered offensive in the same way as the word "ret*d" -- once used for the intellectually challenged as a mere descriptor, but now used so often as an insult that its meaning has lost all neutrality and is now offensive.
I would guess that the person who said you're being treated like an invalid meant that he thinks you're being treated like you're unwell, delicate, needing to be treated like someone who has an illness physically or otherwise, instead of a robust, active person who can take part in everything. Just my guess, as it used to be that an invalid was an unwell person who needed care, usually physical.