The Rise and Fall of Peer Review

Post Reply New Topic

https://www.experimental-history.com/p/ ... eer-review

For the last 60 years or so, science has been running an experiment on itself. The experimental design wasn’t great; there was no randomization and no control group. Nobody was in charge, exactly, and nobody was really taking consistent measurements. And yet it was the most massive experiment ever run, and it included every scientist on Earth.

Most of those folks didn’t even realize they were in an experiment. Many of them, including me, weren’t born when the experiment started. If we had noticed what was going on, maybe we would have demanded a basic level of scientific rigor. Maybe nobody objected because the hypothesis seemed so obviously true: science will be better off if we have someone check every paper and reject the ones that don’t pass muster. They called it “peer review.”

This was a massive change. From antiquity to modernity, scientists wrote letters and circulated monographs, and the main barriers stopping them from communicating their findings were the cost of paper, postage, or a printing press, or on rare occasions, the cost of a visit from the Catholic Church. Scientific journals appeared in the 1600s, but they operated more like magazines or newsletters, and their processes of picking articles ranged from “we print whatever we get” to “the editor asks his friend what he thinks” to “the whole society votes.” Sometimes journals couldn’t get enough papers to publish, so editors had to go around begging their friends to submit manuscripts, or fill the space themselves. Scientific publishing remained a hodgepodge for centuries.

(Only one of Einstein’s papers was ever peer-reviewed, by the way, and he was so surprised and upset that he published his paper in a different journal instead.)

That all changed after World War II. Governments poured funding into research, and they convened “peer reviewers” to ensure they weren’t wasting their money on foolish proposals. That funding turned into a deluge of papers, and journals that previously struggled to fill their pages now struggled to pick which articles to print. Reviewing papers before publication, which was “quite rare” until the 1960s, became much more common. Then it became universal.

Now pretty much every journal uses outside experts to vet papers, and papers that don’t please reviewers get rejected. You can still write to your friends about your findings, but hiring committees and grant agencies act as if the only science that exists is the stuff published in peer-reviewed journals. This is the grand experiment we’ve been running for six decades.

The results are in. It failed.

...

What went wrong?

Here’s a simple question: does peer review actually do the thing it’s supposed to do? Does it catch bad research and prevent it from being published?

It doesn’t. Scientists have run studies where they deliberately add errors to papers, send them out to reviewers, and simply count how many errors the reviewers catch. Reviewers are pretty awful at this. In this study reviewers caught 30% of the major flaws, in this study they caught 25%, and in this study they caught 29%. These were critical issues, like “the paper claims to be a randomized controlled trial but it isn’t” and “when you look at the graphs, it’s pretty clear there’s no effect” and “the authors draw conclusions that are totally unsupported by the data.” Reviewers mostly didn’t notice.

In fact, we’ve got knock-down, real-world data that peer review doesn’t work: fraudulent papers get published all the time. If reviewers were doing their job, we’d hear lots of stories like “Professor Cornelius von Fraud was fired today after trying to submit a fake paper to a scientific journal.” But we never hear stories like that. Instead, pretty much every story about fraud begins with the paper passing review and being published. Only later does some good Samaritan—often someone in the author’s own lab!—notice something weird and decide to investigate. That’s what happened with this paper about dishonesty that clearly has fake data (ironic), these guys who have published dozens or even hundreds of fraudulent papers, and this debacle:

Why don’t reviewers catch basic errors and blatant fraud? One reason is that they almost never look at the data behind the papers they review, which is exactly where the errors and fraud are most likely to be. In fact, most journals don’t require you to make your data public at all. You’re supposed to provide them “on request,” but most people don’t. That’s how we’ve ended up in sitcom-esque situations like ~20% of genetics papers having totally useless data because Excel autocorrected the names of genes into months and years.

...

The Scam

Academics publish s**t.

Some glance at the abstract
To add a reference.

Courtesy.
Reciprocation.

A high citation count
Equals a High Impact.
Elevated prestige.

Peer-review has existed since at least Newton's time and it's actually pretty crucial for scientific research to work. Sure, there may be some cases where reviewers miss things but those should also be weeded out in time with repeated experiments and follow-up critiques.

One problem with modern peer review is the publication for pay companies that have popped up overseas. Most academic tenure-track jobs in the US require publications as part of the tenure review process. These companies will publish absolute garbage articles for a hefty fee. There is no peer review that is done by them.

Those professors who are failing to get their research published by the peer reviewed process can now take a shortcut to publication. Tenure review committees might think that the articles are peer reviewed, as they have titles similar to known journals. I know of a head of a chemistry department who had one published that way just to pad his resume for the administration review. The department even paid the $3500 to do it because he had control of the funds. He should have been fired on the spot for academic dishonesty in my opinion.

I have been contacted via email multiple times over the years by different foreign publishing companies as an attempt to get me to purchase the ability to publish. I delete their message and block the sender. This is a fraud that needs to be stopped. I will not publish my research unless it is done under the correct ethical way.

It's pretty clear now that the modern incarnation of peer review barely works and it may never have worked, it isn't just some cases as you politely put it. The grumblings that I started hearing 15-ish years ago about the state of modern science, its abusive relationship with Truth and its subordinate relationship to politics have become rather more shored up and better investigated. Reviewers, when tested, miss most glaring errors (70%+ failure rate!) and more often than not have no access to the data needed to even make a proper judgement.

Study after study, prank after prank and fraud after fraud have shown that the label of "Peer-reviewed" when applied to science holds about as much value as "Astrologer-approved".

Pair this with the replication crisis and it's looking like modern science going back decades could be 50%+ faked, fiddled or otherwise worthless garbage. If that wasn't bad enough, such rot has a way of spreading - even the well-conducted science might have built upon and used results, theories and structural assumptions from garbage peer-reviewed papers, thus making their work worthless too, despite the best intentions of the authors.



I'm curious. Does your research rely on the published results of others at any point? You might want to double check those results if so, you just can't assume good science any more.

The only part it relies on from the literature is the basic starting point of the reagents I use to form the materials I am researching with. The literature gave clues on which catalyst I needed to work with, so that was helpful in so many ways. I can order those from a chemical company, but prefer to make them from the raw elements themselves. Yes, it takes longer to do that way. I am very patient in my work. I do not need their data other than that.

The reason why I do this long route is because I want to prove beyond a shadow of a doubt that my process works at the molecular level. SEM scans, EPR scans plus the magnetic data I measured confirm my results of the products. I am also developing an entirely new set of materials that have never been reported in the literature. All I can say it that it should raise the Curie temperature of that material by several orders of magnitude. That would change the rules of the game on what I can do with it.

You had better not be messing with anti-gravity and spinning superconductors scientists tend to mysteriously disappear shortly after they report working on that.

The peer-review publication scam is one of the themes of my short story.

https://losinger.altervista.org/documen ... nomist.pdf

No, I am not working in that area. I am after something else that is a better use of my time. I will let someone else play with those theories.