Peer-review and publication does not guarantee reliable info

Page 1 of 2 [ 25 posts ]  Go to page 1, 2  Next

Pepe
Veteran
Veteran

User avatar

Joined: 11 Jun 2013
Age: 53
Gender: Male
Posts: 1,705
Location: Oz

23 Mar 2019, 9:03 pm

Quote:
Peer-review and publication does not guarantee reliable information.
Is there bias associated with peer-review?
Peer-review is by no means perfect. It is itself subject to bias, as most things in research are. Evidence from a peer-reviewed article does not make it reliable, based only on that fact.

For example, there is evidence suggesting poor interrater agreement among peer-reviewers, with a strong bias against manuscripts that report results against reviewers’ theoretical perspectives [5]. Although a study reported in the Journal of General Internal Medicine showed that reviewers agreed barely beyond chance on recommendations to accept/revise vs. reject, editors nevertheless placed considerable weight on reviewer recommendations [6]. In addition, it has been shown that large numbers of public reviews are more thorough in reviewing academic articles than a small group of experts [7].

There is also ongoing debate about reviewer bias in the single-blind peer review process. Some suggest that if reviewers know the identity of authors, there may be implicit bias against women [8] and those with foreign last names or from less prestigious institutions [9]. Therefore, some researchers argue that double-blind peer review is preferable [10].

In addition, some argue that, for multidisciplinary articles, it is difficult to recruit reviewers who are well-versed in all the relevant methodologies since they which tend to cover multiple different topics in a single study, which works against authors of such papers [3]. https://www.students4bestevidence.net/p ... formation/


_________________
I'm not here to change the world...There isn't a big enough soiled nappy bin... ;)
Autistic/scout motto: "Give me a better argument and I will listen..."
"Hate me for who I am...Not for who you think I am..."
You have been weighed...you have been measured...and you have been found wanking...:P
"Honesty is not a social duty, not a sacrifice for the sake of others, but the most profoundly selfish virtue man can practice: his refusal to sacrifice the reality of his own existence to the deluded consciousness of others."
Truth may be inconvenient but it is never politically incorrect...The Oracle of Truth has spoken...8) I'm a rationalist...Deal with it...:mrgreen:


Pepe
Veteran
Veteran

User avatar

Joined: 11 Jun 2013
Age: 53
Gender: Male
Posts: 1,705
Location: Oz

23 Mar 2019, 11:33 pm

Quote:
Let's stop pretending peer review works.

It can take years, multiple sets of fresh eyes, and people with adversarial views for the truth to come to light. Look no further than the study that linked autism to the measles-mumps-rubella vaccine, published in the Lancet. That paper was retracted after it was found to be not only fraudulent but also deeply flawed. https://www.vox.com/2015/12/7/9865086/p ... e-problems


_________________
I'm not here to change the world...There isn't a big enough soiled nappy bin... ;)
Autistic/scout motto: "Give me a better argument and I will listen..."
"Hate me for who I am...Not for who you think I am..."
You have been weighed...you have been measured...and you have been found wanking...:P
"Honesty is not a social duty, not a sacrifice for the sake of others, but the most profoundly selfish virtue man can practice: his refusal to sacrifice the reality of his own existence to the deluded consciousness of others."
Truth may be inconvenient but it is never politically incorrect...The Oracle of Truth has spoken...8) I'm a rationalist...Deal with it...:mrgreen:


Pepe
Veteran
Veteran

User avatar

Joined: 11 Jun 2013
Age: 53
Gender: Male
Posts: 1,705
Location: Oz

23 Mar 2019, 11:59 pm

Quote:
Richard Smith, former Editor‐in‐Chief of the British Medical Journal, blames the peer review system, describing it as “slow, expensive, profligate of academic time, highly subjective, something of a lottery, prone to bias, and easily abused” 2. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4693514/


_________________
I'm not here to change the world...There isn't a big enough soiled nappy bin... ;)
Autistic/scout motto: "Give me a better argument and I will listen..."
"Hate me for who I am...Not for who you think I am..."
You have been weighed...you have been measured...and you have been found wanking...:P
"Honesty is not a social duty, not a sacrifice for the sake of others, but the most profoundly selfish virtue man can practice: his refusal to sacrifice the reality of his own existence to the deluded consciousness of others."
Truth may be inconvenient but it is never politically incorrect...The Oracle of Truth has spoken...8) I'm a rationalist...Deal with it...:mrgreen:


Pepe
Veteran
Veteran

User avatar

Joined: 11 Jun 2013
Age: 53
Gender: Male
Posts: 1,705
Location: Oz

Yesterday, 12:12 am

Quote:
The Australian Historical Association expresses in the strongest terms its staunch opposition to political interference in the peer review process of awarding Australian Research Council grants. https://www.theaha.org.au/aha-response- ... il-grants/


_________________
I'm not here to change the world...There isn't a big enough soiled nappy bin... ;)
Autistic/scout motto: "Give me a better argument and I will listen..."
"Hate me for who I am...Not for who you think I am..."
You have been weighed...you have been measured...and you have been found wanking...:P
"Honesty is not a social duty, not a sacrifice for the sake of others, but the most profoundly selfish virtue man can practice: his refusal to sacrifice the reality of his own existence to the deluded consciousness of others."
Truth may be inconvenient but it is never politically incorrect...The Oracle of Truth has spoken...8) I'm a rationalist...Deal with it...:mrgreen:


Darmok
Veteran
Veteran

Joined: 18 Dec 2015
Gender: Male
Posts: 6,980
Location: New England

Yesterday, 12:57 am

It's interesting how peer review, as an ordinary internal practice in publishing, became politicized by people eager to advance various policy agendas. This routine practice, which no one ever claimed makes papers "true" or "proven," somehow became a stamp of approval (when touted by politicians). "You can't challenge these results — they have been peer reviewed, which means they are true." Nonsense.

My background is in science and I've been a peer reviewer and had papers peer reviewed. It's no different than giving a draft of your paper to the guy down the hall and asking him to look it over and offer suggestions and corrections. Peer review exists mainly because journals traditionally have limited space: there is room for 10 papers in each issue, and they get 20, so the editors have to find some basis for selecting 10 and rejecting 10. They send the MSS out to other specialists for an opinion, and generally follow that opinion. It's a system for journal filtering, not a system for truth-discovery.


_________________
 
There Are Four Lights!


Pepe
Veteran
Veteran

User avatar

Joined: 11 Jun 2013
Age: 53
Gender: Male
Posts: 1,705
Location: Oz

Yesterday, 2:53 am

Darmok wrote:
It's interesting how peer review, as an ordinary internal practice in publishing, became politicized by people eager to advance various policy agendas. This routine practice, which no one ever claimed makes papers "true" or "proven," somehow became a stamp of approval (when touted by politicians). "You can't challenge these results — they have been peer reviewed, which means they are true." Nonsense.

My background is in science and I've been a peer reviewer and had papers peer reviewed. It's no different than giving a draft of your paper to the guy down the hall and asking him to look it over and offer suggestions and corrections. Peer review exists mainly because journals traditionally have limited space: there is room for 10 papers in each issue, and they get 20, so the editors have to find some basis for selecting 10 and rejecting 10. They send the MSS out to other specialists for an opinion, and generally follow that opinion. It's a system for journal filtering, not a system for truth-discovery.


And yet so many people see the peer review system as infallible...
My intention is not to be disrespectful, but there is a lot of ignorance out there in the general community...
And yes, in some cases this naivety is cynically used by those with a political agenda...

What I find "interesting" is how "upset" some people get when this is pointed out to them...
I suspect it primarily has something to do with the uncomfortable feelings associated with uncertainty...
Cognitive dissonance is a biatch...

The "funny" thing is that no one is saying: "You need to factor in this added complexity, and when you do, you will change your original position"...
To the contrary, embracing in-depth critical thinking may reinforce the original belief with the benefit of a more rational basis for one's opinion...

Yes, the process of in-depth investigation is laborious, but what is the alternative?
The embracing of heuristics?
Blind faith in an authority figure or group?
Engaging in pedestrian groupthink?
Or a conceptual crapshoot for the lazy gamblers amongst us?


_________________
I'm not here to change the world...There isn't a big enough soiled nappy bin... ;)
Autistic/scout motto: "Give me a better argument and I will listen..."
"Hate me for who I am...Not for who you think I am..."
You have been weighed...you have been measured...and you have been found wanking...:P
"Honesty is not a social duty, not a sacrifice for the sake of others, but the most profoundly selfish virtue man can practice: his refusal to sacrifice the reality of his own existence to the deluded consciousness of others."
Truth may be inconvenient but it is never politically incorrect...The Oracle of Truth has spoken...8) I'm a rationalist...Deal with it...:mrgreen:


Darmok
Veteran
Veteran

Joined: 18 Dec 2015
Gender: Male
Posts: 6,980
Location: New England

Yesterday, 3:02 am

Here's another valuable website to follow:

https://retractionwatch.com/

They monitor formal retractions and corrections to published scientific/academic papers, because of errors or in some cases fraud (lots of that coming from China these days).


_________________
 
There Are Four Lights!


Pepe
Veteran
Veteran

User avatar

Joined: 11 Jun 2013
Age: 53
Gender: Male
Posts: 1,705
Location: Oz

Yesterday, 3:13 am

Darmok wrote:
Here's another valuable website to follow:

https://retractionwatch.com/

They monitor formal retractions and corrections to published scientific/academic papers, because of errors or in some cases fraud (lots of that coming from China these days).


Much appreciated...


_________________
I'm not here to change the world...There isn't a big enough soiled nappy bin... ;)
Autistic/scout motto: "Give me a better argument and I will listen..."
"Hate me for who I am...Not for who you think I am..."
You have been weighed...you have been measured...and you have been found wanking...:P
"Honesty is not a social duty, not a sacrifice for the sake of others, but the most profoundly selfish virtue man can practice: his refusal to sacrifice the reality of his own existence to the deluded consciousness of others."
Truth may be inconvenient but it is never politically incorrect...The Oracle of Truth has spoken...8) I'm a rationalist...Deal with it...:mrgreen:


cyberdad
Veteran
Veteran

User avatar

Joined: 21 Feb 2011
Age: 51
Gender: Male
Posts: 7,264

Yesterday, 4:00 am

Darmok wrote:
Peer review exists mainly because journals traditionally have limited space: there is room for 10 papers in each issue, and they get 20, so the editors have to find some basis for selecting 10 and rejecting 10. They send the MSS out to other specialists for an opinion, and generally follow that opinion. It's a system for journal filtering, not a system for truth-discovery.


Interesting perspective! Back when I used to be a scientist around 25 years ago I also found journal editors to have subjective criteria over what constitutes "publishable" material. To be fair though, highly cited journals get dumped with 1000s of manuscripts for each edition from around the world. They use various tricks to ignore papers ranging from lack of English translation to the topic not aligning with the journal's research focus (whatever they decide is their favourite pet subject).



SocOfAutism
Veteran
Veteran

Joined: 2 Mar 2015
Gender: Female
Posts: 2,399

Yesterday, 8:26 am

Oh wow! His eyes open wide!

Yes, this is why I did not bother to try to publish my own work. A big part in my working hiatus.

There is a publishing circle jerk (sorry for the term) going on in academia. It got uncomfortable when autism was introduced. The funding for autism all comes from big agencies trying to identify and cure autism, so there is no money for positive autism research. Because of this, there are almost no autism researchers trying to publish about autism in a positive light. It would threaten the narrative of the people trying to cure it.

Many academics are autistic themselves, but in the closet. Coming out would put them in danger of being bullied by their colleagues, students, put them at risk for losing tenor, power, good projects, etc. They won’t do it. Autistic people in high academic positions would only speak to me anonymously. They would not back me.

And yeah, no one ever checks behind you with your findings. I could have made up everything my subjects said. Changed things if I wanted. I had to fight the “human rights board” at my university because they didn’t think autistics we’re capable of speaking for themselves. Some of this bullsh*t CAME FROM autistic people’s mouths, I can only assume because they were playing along, trying to blend in. Like how the gayest person in the world might say the most homophobic things.

It really turned my stomach, made me think less of humanity.



shlaifu
Veteran
Veteran

User avatar

Joined: 26 May 2014
Age: 35
Gender: Male
Posts: 1,095

Yesterday, 8:33 am

What amuses me is that the humanities were well aware of that, decades ago. But the "hard" sciences refused sociologists' comments on institutionalized biases. Then the science wars happened, and there's still scientists occasionally publishing joke papers under fake names in humanities journals - to discredit the humanities.
But slowly, the STEM fields are acknowledging that, indeed, the humanities saw this coming.


_________________
I can read facial expressions. I did the test.


Antrax
Pileated woodpecker
Pileated woodpecker

Joined: 23 Feb 2019
Gender: Male
Posts: 177
Location: Midwest

Yesterday, 10:50 am

As a graduate researcher (biological engineering), peer review is certainly not perfect by any means. But when you're operating at the fringes of human understanding there really is no other alternative. No objective criteria exists for new knowledge.



Fnord
Veteran
Veteran

User avatar

Joined: 6 May 2008
Gender: Male
Posts: 35,704
Location: Stendec

Yesterday, 10:56 am

Pepe wrote:
Peer-Review And Publication Does Not Guarantee Reliable Info.
Maybe, but it is magnitudes more reliable than self-published “research” or “research” that is first announced in a tabloid.

Why do you want to discredit science?



The_Walrus
Forum Moderator
Forum Moderator

User avatar

Joined: 27 Jan 2010
Age: 24
Gender: Male
Posts: 6,073
Location: Reading, England

Yesterday, 1:20 pm

I don't think sensible people claim that peer-review is infallible, but peer-reviewed work is certainly much more reliable than non-reviewed work.



Pepe
Veteran
Veteran

User avatar

Joined: 11 Jun 2013
Age: 53
Gender: Male
Posts: 1,705
Location: Oz

Yesterday, 4:50 pm

SocOfAutism wrote:

Because of this, there are almost no autism researchers trying to publish about autism in a positive light. It would threaten the narrative of the people trying to cure it.


Doesn't surprise me in the least...


_________________
I'm not here to change the world...There isn't a big enough soiled nappy bin... ;)
Autistic/scout motto: "Give me a better argument and I will listen..."
"Hate me for who I am...Not for who you think I am..."
You have been weighed...you have been measured...and you have been found wanking...:P
"Honesty is not a social duty, not a sacrifice for the sake of others, but the most profoundly selfish virtue man can practice: his refusal to sacrifice the reality of his own existence to the deluded consciousness of others."
Truth may be inconvenient but it is never politically incorrect...The Oracle of Truth has spoken...8) I'm a rationalist...Deal with it...:mrgreen: