91社区

Cancer scientists overestimate the extent to which high-profile preclinical studies can be successfully replicated, new research from 91社区 suggests.

The聽findings, published in PLOS Biology by Jonathan Kimmelman and colleagues from 91社区, are based on a survey in which both experts and novices were asked to predict whether mouse experiments in six prominent preclinical cancer studies conducted by the (RP:CB) would reproduce the effects observed in original studies.

On average, the researchers forecasted a 75% probability of replicating statistical significance, and a 50% probability of reproducing the same size effect as in the original study. Yet according to these criteria, none of the six studies already completed by the Reproducibility Project showed the same results previously reported.

One possible explanation for the optimism is that cancer scientists overestimate the replicability of major reports in their field. Another is that they underestimate the logistical and methodological complexity of independent laboratories repeating these techniques.

Reproducibility crisis

The work follows on numerous reports exploring biomedicine鈥檚 so-called reproducibility crisis. In the last 10 or 15 years, there have been mounting concerns that some of the techniques and practices used in biomedical research lead to inaccurate assessments of a drug鈥檚 clinical promise.

Given that not all studies reproduce, Kimmelman and his team wondered if cancer experts could at least sniff out which studies would not easily replicate. The finding that cancer researchers鈥� ability to do so 鈥渨as really limited鈥� suggests that there may be inefficiencies in the process by which science 鈥渟elf-corrects.鈥�

There is however strong community concern that, due to process-related issues and potential methodological differences, the replication studies themselves may not be an entirely reliable measure of replication outcome. Kimmelman emphasizes that the findings don鈥檛 indicate that scientists who participated in the study don鈥檛 understand what鈥檚 going on their field 鈥� nor does it diminish the importance of funding research and making policy on the basis of scientific consensus. Some scientists were highly accurate in their predictions, and participants were new to forecasting, which is difficult.

Training could be part of the solution

The results do, however, raise the possibility that training might help many scientists overcome certain cognitive biases that affect their interpretation of scientific reports.

鈥淚f the research community believes a finding to be reliable, it might start building on that finding only to later discover the foundations are rotten. If scientists suspect a claim to be spurious, they are more likely to test that claim directly before building on it.鈥�

鈥淭his is the first study of its type, but it warrants further investigation to understand how scientists interpret major reports,鈥� Kimmelman says. 鈥淚 think there is probably good reason to think that some of the problems we have in science are not because people are sloppy at the bench, but because there is room for improvement in the way they interpret findings.鈥�

__________________________________________________________________________

The funding for this study was provided by the Canadian Institutes for Health Research

, by Daniel Benjamin, David R. Mandel, Jonathan Kimmelman, PLOS Biology

Photo credit:聽Cancer Institute of Columbia