Methodological issues plague studies of early autism

Post Reply New Topic

Methodological issues have plagued studies on the effectiveness of early autism interventions for nearly three decades, according to new unpublished research. These problems include multiple types of bias and an overreliance on caregivers to report outcomes.

Researchers presented the findings virtually yesterday at the 2021 International Society for Autism Research annual meeting. (Links to abstracts may work only for registered conference attendees.)

“Poor-quality studies tend to inflate our estimates of intervention effectiveness,” says lead researcher Micheal Sandbank, assistant professor of special education at the University of Texas at Austin.

The new work aimed to evaluate the reliability of studies that examine the effectiveness of early interventions. Conventional wisdom on therapy says “the earlier the better, and the more the better,” Sandbank says.

But the evidence supporting that idea is weak, she and her colleagues found. Several sources of bias threaten to undermine the reliability of many early-intervention studies: selection bias, when experimental groups and control groups are not randomly assigned; detection bias, when the same person administers an intervention and judges its effectiveness; and attrition bias, when participants from a control or experimental group disproportionately drop out over time.

Many studies also leaned heavily on reports from parents or caregivers to assess the effectiveness of an intervention, instead of relying on clinical observations, which studies show are more reliable. Between 25 and 50 percent of the studies from 2011 to 2017 relied on parent reports, down from around 90 percent in 2000.

The team’s 2020 meta-analysis concluded that many early interventions had positive effects. But once they excluded studies that relied on caregiver reports or had a high risk of one or more types of bias, there were often too few studies left to assess an intervention’s effects.

... once they excluded studies that relied on caregiver reports or had a high risk of one or more types of bias, there were often too few studies left to assess an intervention's effects.

All psychiatric and psychological research relies on personal opinion.  Typically there will be ratings from teachers or clinicians based on absence, presence, or frequency of specific observable behaviors.  The problem comes when the person doing the rating or responding has a dog in the fight.  If it's the person running the study, they have a bias towards certain outcomes.  What they would need is a completely independent observer...

It's not easy, that's for sure....and it would invite criticism, because "functioning levels" would come into play.

One example is the oft-stated claim that 80 or 85% of people with autism are unemployed. One wonders whether all autistic folks were included in this determination. One wonders, if only "high-functioning" folks were taken into consideration, whether the 80 or 85% figure would hold up.

The academic and governmental/NGO take on autistic tends to overlook the fact that many adult autistics do work, or how many of them may be flying under the diagnostic radar (never were diagnosed as kids; and are still undiagnosed)

Usually, researchers don't take into account the "spectrum" nature of autism.