Methodological issues plague studies of early autism

Page 1 of 1 [ 13 posts ] 

ASPartOfMe
Veteran
Veteran

User avatar

Joined: 25 Aug 2013
Age: 64
Gender: Male
Posts: 25,888
Location: Long Island, New York

07 May 2021, 5:19 am

Spectrum

Quote:
Methodological issues have plagued studies on the effectiveness of early autism interventions for nearly three decades, according to new unpublished research. These problems include multiple types of bias and an overreliance on caregivers to report outcomes.

Researchers presented the findings virtually yesterday at the 2021 International Society for Autism Research annual meeting. (Links to abstracts may work only for registered conference attendees.)

“Poor-quality studies tend to inflate our estimates of intervention effectiveness,” says lead researcher Micheal Sandbank, assistant professor of special education at the University of Texas at Austin.

The new work aimed to evaluate the reliability of studies that examine the effectiveness of early interventions. Conventional wisdom on therapy says “the earlier the better, and the more the better,” Sandbank says.

But the evidence supporting that idea is weak, she and her colleagues found. Several sources of bias threaten to undermine the reliability of many early-intervention studies: selection bias, when experimental groups and control groups are not randomly assigned; detection bias, when the same person administers an intervention and judges its effectiveness; and attrition bias, when participants from a control or experimental group disproportionately drop out over time.

Many studies also leaned heavily on reports from parents or caregivers to assess the effectiveness of an intervention, instead of relying on clinical observations, which studies show are more reliable. Between 25 and 50 percent of the studies from 2011 to 2017 relied on parent reports, down from around 90 percent in 2000.

The team’s 2020 meta-analysis concluded that many early interventions had positive effects. But once they excluded studies that relied on caregiver reports or had a high risk of one or more types of bias, there were often too few studies left to assess an intervention’s effects.


_________________
Professionally Identified and joined WP August 26, 2013
DSM 5: Autism Spectrum Disorder, DSM IV: Aspergers Moderate Severity.

“My autism is not a superpower. It also isn’t some kind of god-forsaken, endless fountain of suffering inflicted on my family. It’s just part of who I am as a person”. - Sara Luterman


Jiheisho
Veteran
Veteran

Joined: 21 Jul 2020
Age: 57
Gender: Male
Posts: 1,507

07 May 2021, 7:31 am

Those are all common problems. I am not really surprised.



BeaArthur
Veteran
Veteran

Joined: 11 Aug 2015
Posts: 5,775

07 May 2021, 7:59 am

Well, autistic children are not lab rats. You can't be methodologically pure where human beings are at stake.

I grant you, one well-designed study that attempts to observe and evaluate without parent or treatment-provider bias, could answer a lot of questions. But that's very expensive to do.


_________________
A finger in every pie.


Fnord
Veteran
Veteran

User avatar

Joined: 6 May 2008
Age: 64
Gender: Male
Posts: 50,581
Location: Stendec

07 May 2021, 8:30 am

Quote:
... once they excluded studies that relied on caregiver reports or had a high risk of one or more types of bias, there were often too few studies left to assess an intervention's effects.
Once they dismissed reports that relied on personal opinions, they were left with too little to reach a conclusion.

Objective data is essential to valid research.


_________________
 Link to Official List of Trump's Atrocities 

45OFFICE = TRE45ON
Lock Him Up!


BeaArthur
Veteran
Veteran

Joined: 11 Aug 2015
Posts: 5,775

07 May 2021, 12:40 pm

All psychiatric and psychological research relies on personal opinion. Typically there will be ratings from teachers or clinicians based on absence, presence, or frequency of specific observable behaviors. The problem comes when the person doing the rating or responding has a dog in the fight. If it's the person running the study, they have a bias towards certain outcomes. What they would need is a completely independent observer, but that's expensive, especially over a large N (sample size) and an extended period of time.

The answer is not to throw out all the research, but to be aware of these necessary biases and try to reduce them as much as possible. Don't throw out the baby with the bath water, so to speak. Save the bath water for watering plants with, and put the baby back in the basinette.


_________________
A finger in every pie.


kraftiekortie
Veteran
Veteran

User avatar

Joined: 4 Feb 2014
Gender: Male
Posts: 77,148
Location: Queens, NYC

07 May 2021, 12:46 pm

Usually, researchers don't take into account the "spectrum" nature of autism.



BeaArthur
Veteran
Veteran

Joined: 11 Aug 2015
Posts: 5,775

07 May 2021, 1:15 pm

I don't understand your objection, KK. How would they do that?


_________________
A finger in every pie.


kraftiekortie
Veteran
Veteran

User avatar

Joined: 4 Feb 2014
Gender: Male
Posts: 77,148
Location: Queens, NYC

07 May 2021, 1:26 pm

It's not easy, that's for sure....and it would invite criticism, because "functioning levels" would come into play.

One example is the oft-stated claim that 80 or 85% of people with autism are unemployed. One wonders whether all autistic folks were included in this determination. One wonders, if only "high-functioning" folks were taken into consideration, whether the 80 or 85% figure would hold up.



Fnord
Veteran
Veteran

User avatar

Joined: 6 May 2008
Age: 64
Gender: Male
Posts: 50,581
Location: Stendec

07 May 2021, 1:32 pm

BeaArthur wrote:
All psychiatric and psychological research relies on personal opinion.  Typically there will be ratings from teachers or clinicians based on absence, presence, or frequency of specific observable behaviors.  The problem comes when the person doing the rating or responding has a dog in the fight.  If it's the person running the study, they have a bias towards certain outcomes.  What they would need is a completely independent observer...
Okay, I'll buy that.


_________________
 Link to Official List of Trump's Atrocities 

45OFFICE = TRE45ON
Lock Him Up!


BeaArthur
Veteran
Veteran

Joined: 11 Aug 2015
Posts: 5,775

07 May 2021, 1:56 pm

kraftiekortie wrote:
It's not easy, that's for sure....and it would invite criticism, because "functioning levels" would come into play.

One example is the oft-stated claim that 80 or 85% of people with autism are unemployed. One wonders whether all autistic folks were included in this determination. One wonders, if only "high-functioning" folks were taken into consideration, whether the 80 or 85% figure would hold up.

Yeah, that's a pet peeve of mine too. Let's assume that figure is wrong. It may unnecessarily lead to some autistic folks not trying at all, and to some employers believing that most autistic people are unemployable so why should they take a chance.

I guess this is something that will need to improve as the subject area matures. The academic and governmental/NGO take on autistic tends to overlook the fact that many adult autistics do work, or how many of them may be flying under the diagnostic radar (never were diagnosed as kids; and are still undiagnosed)


_________________
A finger in every pie.


kraftiekortie
Veteran
Veteran

User avatar

Joined: 4 Feb 2014
Gender: Male
Posts: 77,148
Location: Queens, NYC

07 May 2021, 2:02 pm

Yep....I think that's one of the "cruxes" of it.....



kitesandtrainsandcats
Veteran
Veteran

User avatar

Joined: 6 May 2016
Age: 58
Gender: Male
Posts: 2,232
Location: Missouri

07 May 2021, 2:03 pm

BeaArthur wrote:
The academic and governmental/NGO take on autistic tends to overlook the fact that many adult autistics do work, or how many of them may be flying under the diagnostic radar (never were diagnosed as kids; and are still undiagnosed)

That group would include me, I was undiagnosed until I was over 40 & worked until physical disabilities finally put an end to my working days around the time I was 43.


_________________
"There are a thousand things that can happen when you go light a rocket engine, and only one of them is good."
Tom Mueller of SpaceX, in Air and Space, Jan. 2011


CockneyRebel
Veteran
Veteran

User avatar

Joined: 17 Jul 2004
Age: 46
Gender: Male
Posts: 104,938
Location: Canada in person, Germany in spirit

07 May 2021, 11:35 pm

kraftiekortie wrote:
Usually, researchers don't take into account the "spectrum" nature of autism.


That's the thing that cheeses me off about researchers. They refuse to see autism as the spectrum that it is. They're living in the 1950s.


_________________
Peabody

Om Nom 2024

viewtopic.php?f=11&t=26&start=645