The Rules of Replication

Zhang is not alone in feeling slighted by the handling of a replication attempt. As part of the large, crowdsourced initiative in social psychology called the Reproducibility Project: Psychology, coordinated by the Charlottesville, Virginia–based Center for Open Science, researchers failed to replicate the findings of a study on cleanliness and people’s moral judgments by Simone Schnall, a senior lecturer at the University of Cambridge in the U.K. As Schnall has described it, tweets, e-mails, and a blog post announcing the failure were broadcast to the field before she had a chance to address the discrepancy. “I feel like a criminal suspect who has no right to a defense and there is no way to win,” Schnall wrote of her experience in a blog post last spring: “The accusations that come with a ‘failed’ replication can do great damage to my reputation, but if I challenge the findings I come across as a ‘sore loser.’”

To avoid such unfair judgments, Nobel prize–winning psychologist Daniel Kahneman earlier this year proposed a new etiquette for replication. He suggested that certain actions could be taken by replicating labs to avoid what he calls adversarial replication. These include contacting the original lab and discussing the protocol; inviting the original author to comment on the proposed replication experiments; discussing any amendments to the protocol; and allowing reviewers to read the correspondence. “The rules are designed to motivate both author and replicator to behave reasonably even when they are thoroughly irritated with each other,” Kahneman wrote in a commentary outlining his suggested guidelines, posted to Scribd in May.

Zhang agrees that Marshall’s replication attempt would have been handled better had the group discussed the experimental details with him.

Kahneman’s model for replication represents just one of many ways labs can go about trying to reproduce the work of others. In fact, some researchers don’t agree that it’s always beneficial to fully involve the original lab in a replication attempt. Marshall at miRagen says his group will sometimes contact the original authors to ask for help if information is missing from the paper, “but we really want to be able to do this completely separately and independently.” For his firm to invest resources into a technology or a procedure, Marshall wants to be reassured that the findings are robust enough to hold up in an independent lab.

A sister effort to the Reproducibility Project: Psychology, called the Reproducibility Project: Cancer Biology, will soon begin to roll out the results from a massive set of replication attempts. The largest coordinated effort of its kind, the cancer project is attempting to redo the main experiments from the 50 most-cited papers in cancer biology from 2010 to 2012. Over the coming months, results from the project will be published in e Life.

The Reproducibility Project: Cancer Biology operates similarly to Kahneman’s proposal with regard to the original authors’ participation. In each case, the replication protocol goes through peer review before the experiments ever start, and an original author is always a reviewer. After the work is complete, the data are posted, and the final manuscript is again peer-reviewed before publication.

“We’re completely open and transparent,” says Errington. In addition to identifying potential discrepancies in the protocol, Errington says, it’s essential to get the original authors on board to source materials necessary for the experiments. For instance, reagents, cell lines, or animal models may have been custom-made for the project by the original lab, and without access to them, the project could hit a wall.

Be the first to comment

Leave a Reply

Your email address will not be published.