Line of octahedrons banner image

Two public symposia on the many faces of reproducibility with invited speakers from outside the university will be held over the course of the project.

Frontiers of Reproducibility 

October 19, 2020 4 pm - 6:30 pm. This event will be held online.

Replicability and Beyond: Reimagining Science as Truly Open

Alison LedgerwoodUniversity of California Davis

In recent years, a reform movement has emerged across multiple scientific disciplines seeking to improve the quality of our scientific methods and practices, with a focus on enhancing replicability and transparency. I will discuss how efforts to advance replicability and transparency align with—and in fact, cannot succeed in the absence of—efforts to advance generalizability and inclusiveness in science. I will describe how we can change incentive structures to achieve these goals and how the pandemic affords a unique opportunity to do so.

Ego Depletion: A Case Study in Large-Scale Replication

Kathleen VohsUniversity of Minnesota

This talk will discuss a large-scale replication of an ego depletion effect, which involved 36 laboratories from 9 countries and more than 3500 participants. It tested the hypothesis that using self-control on an initial task would render subsequent self-control less successful than if not deployed earlier. The study used a novel design called the paradigmatic replication approach, followed open science practices and introduced new ones as well. The talk will review the study itself, the novel replication model, the open science choices employed, and the aims and goals thereof.

Should Science be 100% Replicable?

Wendy Wood, University of Southern California

Contrary to much current wisdom, the answer is clearly, “no.” Efforts to make science completely replicable fail to distinguish reproducibility from replicability. Reproducibility refers to getting consistent results from reanalyzing the same data using the same computations and methods of analysis (NASEM, 2019). Yes, complete reproducibility is necessary for reliable scientific knowledge. However, replicability refers to consistent results across studies using different data to test the same scientific question. The replication crisis discourse often overlooks that non-replicability is a normal part of the scientific process and can advance scientific knowledge. Nature is intrinsically varied and complex. Non-replicability can be helpful in identifying limits in current scientific knowledge and technology. I illustrate these claims with failures to replicate that led to the discovery of new phenomena and new insights about variability in established phenomena. The implication for science policy is to set standards that allow for helpful forms of non-replicability.

The Social Limits of Knowledge

James EvansUniversity of Chicago

The explosive growth of scientists, scientific journals, articles and findings in recent years has exponentially increased the difficulty scientists face in navigating prior knowledge and collectively reasoning over it to drive future advance. This challenge is exacerbated by uncertainty about the reproducibility of published findings. Here I detail a series of investigations using large-scale publication, experimental and clinical data, which reveal how socially, methodologically and institutionally independent research activity is much more likely to replicate than work performed within a singular community of researchers using the same methods and approaches. These findings recommend policies like decentralized collaboration that go against the common practice of channeling biomedical research funding into nonredundant centralized research consortia and institutes rather than dispersing it more broadly. We show how using this pattern can help to decode bias, predict and improve the replicability of published findings. These findings mesh with other work that demonstrates how densely connected research communities are also associated with a reduction in the rate of discoveries and the likelihood of disruptive advance. Together, these findings highlight the limits of knowledge that form within insulated bubbles of scientific discourse.