“Who are the peers?” (the question nobody asks)
In theory, “peers” means:
- experts in the same field
- with similar training
- capable of evaluating the work
In practice, “peers” often means:
- the same small circle of people
- trained in the same paradigm
- funded by the same agencies
- incentivized to protect the status quo
- reviewing anonymously with no accountability
So the real answer to “Who are the peers?” is:
People who share the same assumptions, incentives, and blind spots as the authors.
That’s not objectivity. That’s in‑group validation.
“What are they reviewing?” (not what people think)
Most people imagine peer review as:
- replicating the experiment
- checking the raw data
- verifying the statistics
- auditing the methods
But that’s not what happens.
Peer reviewers almost never:
- see the raw data
- inspect the code
- reproduce the analysis
- verify the patient records
- check for fraud
- test alternative hypotheses
What they actually review is:
- the manuscript
- the narrative
- the statistical summaries
- the framing
- the conformity to accepted theory
“What are they reviewing?” The honest answer is:
They are reviewing the story, not the science.
Peer review is not a truth filter, it’s a gatekeeping mechanism
This is the part most people don’t understand.
Peer review does not determine whether something is true. It determines whether something is acceptable, conventional (mustn’t buck convention), methodologically orthodox, aligned with the field’s assumptions (can’t step out of line), and not too disruptive (or you’ll be sent to the principal).
It’s a cultural filter, not a reality filter.
And this is why:
- paradigm‑shifting work gets rejected
- replication failures get ignored (oh there’s more on this below)
- negative results go unpublished (the dead are removed from the statistics)
- controversial findings get buried (along with the dead)
- industry‑friendly studies sail through (it’s a breeze!)
Peer review is a social process, not a scientific one. Were it a scientific process, the studies conducted on the peer review process wouldn’t have all failed.
Bias is built into the structure
A. Reviewer anonymity encourages unaccountable bias
People are harsher, more political, and more territorial when their names aren’t attached.
B. Journals have incentives
Prestige journals want novelty, splashy results, positive findings, and big-name authors. Not truth.
C. Funding sources shape outcomes
Reviewers know who funds what. They know which results are “expected.”
Or as we stated very clearly in Studies Show: Money influences outcome.
D. Methodological orthodoxy punishes innovation
Anything that challenges the dominant model gets hammered.
This is why unconventional researchers — Gonzalez, Wakefield, Zamboni, Linus Pauling, even early Barry Marshall — get shredded long before their ideas are evaluated on merit.
But we are getting close
Peer review is evolving.
Open peer review
When names are public and comments are public, accountability increases.
Preprint culture
ArXiv, bioRxiv, and medRxiv are preprint servers. Scientists post research papers on these before they go through peer review. Science is now debated in the open before journals touch it.
Post‑publication review
The real scrutiny now happens after publication, not before.
Data transparency
More journals require raw data, code, and preregistration.
Replication crisis awareness
Now this is where things get sticky. In the last decade entire scientific fields have been forced to face an uncomfortable truth that many of their most cited, most famous and trusted studies just don’t replicate. One thing about science is that an experiment conducted in Switzerland, that is conducted later in Las Vegas with all things being equal, will have equal outcomes. This is a foundation of the scientific principle. Though this principle is true in physics, it’s not reliable in biology, medicine, psychology, or the human-centered sciences.
Fields like psychology, oncology, and nutrition are being forced to confront their failures.
Psychology was the first field to get publicly embarrassed around 2011 to 2015 when researchers were unable to replicate many famous studies. Some estimate the failure rate was 70-89%.
One reason was p-hacking, a term I just learned. You’ve heard that “statistics never lie but liars use statistics.” Well, this is where the lying comes in.
College level statistics is one of those classes everyone dreads. There are a handful, though, who get it. And they get so good with statistics that they can manipulate outcomes to appear better, or worse, than they actually are. They use “statistical tricks” to make things look significant. And that’s called p-hacking.
And then there are sample sizes, publication bias, and flashy results favored over solid ones. Magazine articles are supposed to be read, so publishers make them jump off the page. Finally there’s the pressure to publish novel findings. Those are attention getters. But none of these validate the results of a psychological study.
Oncology was next on the chopping block. Cancer is a scary word. So cancer research is supposed to be the most rigorous research in medicine. Instead, it turned out to be the least rigorous.
In 2012, the biotech company Amgen tried to replicate 53 “landmark” cancer studies. They could reproduce 6 of them. That’s an 89% failure rate.
There are things about studies in biology that should be clear to everyone, especially the public, and one of them is animal models that don’t match human biology. What’s good for the goose might be good for the gander, but what’s good for mice is not often good for humans. And there’s the pressure to publish “breakthroughs!” How can Susan G Komen and the American Society fill their coffers if a cure isn’t right around the corner? In the history of cancer research, my little black book contains names that have been traduced by the charities named above, though they were dedicated and focused on saving lives: Ulric Abel, Warbug, Budwig, Nicholas Gonzales, Prusiner, and Zamboni. These were all outliers who worked according to the French maxim about the true purpose of medicine:
Guérir parfois, soulager souvent, consoler toujours.
To cure sometimes, relieve often, comfort always.
Finally, there’s that one field that is built on sand: Nutrition.
It’s from the Nutritional scientists we’ve gotten: coffee causes cancer, coffee prevents cancer, fat kills you, fat saves you, red meat shortens life, meat eaters live longer.
Then there is the most famous fraud perpetrated on Americans that came from Harvard. Scientists took payola to promote the lipid theory of heart disease, that fats were the culprit, when they knew all along that sugar was the culprit. And that narrative shaped American nutrition for half a century.
Nutritional studies just cannot be replicated. A food-frequency questionnaire is deeply unreliable, as are observational studies, and there are the confounding variables, so, weekly, they published contradictory findings.
Nutrition is the field where replication is almost impossible, because real‑world eating is too complex to control.
Finally
Peer review is not a guarantee of truth. It’s a consensus‑filter run by people who share the same assumptions. They review the manuscript, not the reality behind it. The system is improving, but it has not yet proven itself to be unbiased, only self‑consistent.
