Aug
2
Fixing the Replication Crisis
Are you familiar with Dr. Jay Bhattacharya?
Bhattacharya was a professor of medicine at Stanford University, where he received both an MD and a PhD in economics. He has held positions at the National Bureau of Economic Research, the Stanford Institute for Economic Policy Research, the Freeman Spogli Institute for International Studies, and the Standford Center on the Demography and Economics of Health and Aging. He has been published in economics, statistics, legal, medical, public health, and health policy journals. And, of course, Bhattacharya has most recently served as director of the National Institutes of Health (NIH) since April 2025 and acting director of the CDC since Feb. 2026.
Dr. Bhattacharya gave a speech at Hillsdale College this past April, which was adapted into an article for Hillsdale’s Imprimis newsletter. Bhattacharya presents an historical defense of the value of the NIH while also noting issues in the areas of scientific research and public health management. In particular, he lays out three problems that need to be addressed in order to “Make America Healthy Again” and launch a second scientific revolution.
His comments on Problem #1 — and how the NIH aims to correct the problem on their end — is the topic of the day, and I reproduce that section for you below…
— — —
The first of these problems is the replication crisis. Scientists publish studies today; we hear about their findings on TV, in newspapers, and on podcasts; and we (and even our doctors) take them for granted. But then when other scientists ask the same questions and perform the same experiments, they do not come up with the same results. That means that a lot of the science we (and our doctors) take for granted is not actually true.
Are eggs good for us? I grew up in the 1970s and 1980s, when eggs were considered the next worst thing to poison. I didn’t eat an egg until I was 19, and then it was a tasteless egg white omelet, since only the whites of eggs were considered healthy in the 1990s. But today eggs are considered a superfood! There are so many examples like this, and at the root of the confusion is the replication crisis.
How do we address this? We begin by recognizing the fact that most ideas hatched by scientists do not work or are not true. That is completely normal in science and always has been. John Ioannidis, a former colleague of mine at Stanford and one of the most frequently cited scientists in the world, wrote a paper in 2005 entitled, “Why Most Published Research Findings Are False.” He was correct, and the reason for this is that science is hard. That is why it is frustrating to be a scientist — and why, to be a good scientist, an essential requirement is epistemic humility.
We also need to recognize that just because a study is published in a prestigious journal, it does not mean the study is true — not even if it is peer reviewed! Peer reviewers do not double check the data of a study. They just look at it and then accept or reject it. Peer review is not a measure of scientific truth; it is basically a measure of approval by an editor. It is not meaningless, because presumably the editor has some expertise in the subject. But it is in no way a confirmation of truth.
The only effective way for scientists to check one another is through replication. If I write a paper proposing a theory, I will present the experiments that I used to come up with that theory. If I am right, other scientists should be able to replicate those same experiments and come up with the same answer. That is narrow replication. Even better is reproducibility, by which other people use a different method to come up with the same answer to the question. This is how science operated in the past, and we must return to it if science is to regain the public trust.
The NIH can do a lot to achieve this goal. We can do it first by funding replication. And scientists should welcome this. If someone wants to replicate a study I produced, I should not view that as a threat because it might prove me wrong. I should look on it as an honor because replication is a source of advancements in science. If my study turns out not to be true, then maybe we can figure out why. That kind of collaboration is how science advances.
Let me give you an example of something we are doing already. We funded a project called the Autism Data Science Initiative. The goal is to figure out the causes of the increasing rates of autism. As part of this initiative, we funded replication teams to look at new and promising studies scientists are undertaking to see if they can be replicated.
In addition to funding replication studies, we have to make it easier to discover whether scientific ideas have been replicated. There is a popular database called PubMed that people use to look up biomedical papers. Currently, when you look up a paper, you can see where it was published and cited but not whether it has been replicated. So the NIH is going to add a replication button, and when users click on it, they will be able to see the related replication studies if any.
— — —
Some good reminders in there for those of us on both sides of the political aisle (or somewhere in between).
If you’re interested, the other two problems that Bhattacharya addressed in brief are “Scientific Stagnation” and “Funding Concentration”. To read what he had to say, check out the full article. Btw, I touched on some of these issues in an earlier blogpost, as well.
