I agree with you. For instance, I think simply being referred to as a gay writer, or an Asian artist, or a black painter, or a homeless sculptor, or a short animator, or a woman scientist is a discredit to the person involved. Taken on talent alone, without prejudice, I think one would feel better being known as simply a writer or animator or sculptor.
Of course, we now live in a world where these differences are segmented and publicized as such out in the general marketplace. Some fiction novels that just happen to have an autistic protagonist gets labeled in the Autism novels section. Novels that just so happens to be written by women get lumped in with the Female Novels section. These differences do exist and they can be profitable, but I understand what you mean. Someone simply looking for a good novel to read will ignore books listed in the Autistic Novels section or the Female Novels section or the Black novels section. Go for it anyway. After all, some recognition is better than no recognition at all and, in the future, you could become popular enough to break out of that confined category.