Spring 2020 Research Synthesis (East Bank - Thursday) | The Many Faces of Reproducibility

300 Ford Hall

Meetings will be on Thursdays (approximately every other week) 12:00-1:15 pm.

Spring semester meetings have been cancelled.

Description: Over the past decade there has been growing concern about the replicability of scientific findings. Landmark studies have been discredited; entire fields of inquiry have been shown to rest on shallow foundations. Research syntheses, and especially meta-analysis, have been hailed not only as a way to document replicability but ultimately as a model of cumulative science. In this journal club we will critically evaluate the utility of synthetic approaches, and especially meta-analysis, for addressing the replicability problem.

Structure: There are 6 planned meetings and we anticipate that this will be the first installment in a likely two-part sequence on Research Synthesis and Replicability. The organizers have identified a set of topics and associated readings. We will also solicit topics and suggested readings from participants. To get us going, the organizers will select the first two topics and sets of readings. After that, it will be up to the group to decide on topics/readings. If you would like to suggest topics or readings, please e-mail the organizers: Matt McGue ([email protected]) and Niels Waller ([email protected]).

Tentative Topics:

What does it mean to say that a scientific finding is replicated or replicable?
Research biases: What biases appear to be contributing to the replicability problem?
Types of replication
Types of systematic reviews
Methods for identifying study biases in meta-analyses
Alternative meta-analytic approaches (e.g., cumulative meta-analysis)
Case Study #1: Ego Depletion
Case Study #2: 5-HTT x Life stress, Genotype-environment interaction
Critique of the meta-analytic approach
Alternatives

**Readings for Spring 2020 Research Synthesis**
January 30: Research Synthesis Reading Group Proposal Schmidt, S. (2009). Shall we Really do it Again? The Powerful Concept of Replication is Neglected in the Social Sciences. Review of General Psychology, 13(2), 90–100. Maxwell, S. E., Lau, M. Y., & Howard, G. S. (2015). Is psychology suffering from a replication crisis? What does “failure to replicate” really mean? American Psychologist, 70(6), 487–498.
February 13: Bryan, C. J., Yeager, D. S., & O'Brien, J. M. (2019). Replicator degrees of freedom allow publication of misleading failures to replicate. Proceedings of the National Academy of Sciences of the United States of America, 116(51), 25535-25545. doi: 10.1073/pnas.1910951116. Chan, A. W., Hrobjartsson, A., Haahr, M. T., Gotzsche, P. C., & Altman, D. G. (2004). Empirical evidence for selective reporting of outcomes in randomized trials - Comparison of Protocols to published articles. Jama-Journal of the American Medical Association, 291(20), 2457-2465. doi: 10.1001/jama.291.20.2457. Optional reading: Franco, A., Malhotra, N., & Simonovits, G. (2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345(6203), 1502-1505. doi: 10.1126/science.1255484.
February 27: The practice of research synthesis. Siddaway et al. (2019) provides a detailed discussion of how systematic reviews should be undertaken, including how to avoid common problems. Lakens et al. (2020) show that the recommendations made by Siddaway et al. do not appear to be regularly followed in practice. References: Lakens, D., van Assen, M., Anvari, F. et al. (2020). Examining the reproducibility of meta-analyses in psychology: A preliminary report. Preprint. (pdf) Siddaway, A. P., Wood, A. M., & Hedges, L. V. (2019). How to Do a Systematic Review: A Best Practice Guide for Conducting and Reporting Narrative Reviews, Meta-Analyses, and Meta-Syntheses. In S. T. Fiske (Ed.), Annual Review of Psychology, Vol 70 (Vol. 70, pp. 747-770). Palo Alto: Annual Reviews. (pdf)
March 19: CANCELLED
April 2:
April 16: