Different people believe different things, but saying it would be selfish to have a child knowing it would be special needs is the same as saying that child would have preferred not to be born. I don't know anyone who thinks this way and nothing can be more wrong. Most people with special needs find their lives quite worth living, and do not wish they had never been born.
Have you seriously heard anyone say the things you suggest in your post? I've heard the "I don't think I could handle it" or the "I think it would be unfair to the child" (because of the "I don't think I could handle it" part), but I've never heard anyone say it would be selfish.
And I skipped the test that would have told me something was genetically wrong. I knew it wouldn't have changed anything, and an amnio does pose a risk (very small) to the baby. You don't have to take these tests where there is no medical intervention possible based on the results other than termination.
_________________
Mom to an amazing young adult AS son, plus an also amazing non-AS daughter. Most likely part of the "Broader Autism Phenotype" (some traits).