A lot of new interesting work is being done on questions of reproducibility across the sciences by diverse researchers from different disciplines. Here we share 2-3 papers every month that come from what Project Members are reading. Have a suggestion? Let us know.
July 2022
Mayo, D.G. and D. Hand. 2022. Statistical significance and its critics: practicing damaging science, or damaging scientific practice? Synthese 200:220.
Stewart, S.L.K., Pennington, C.R., da Silva, G.R. et al. 2022. Reforms to improve reproducibility and quality must be coordinated across the research ecosystem: the view from the UKRN Local Network Leads. BMC Research Notes 15:58.
June 2022
Gibson, G. 2022. Perspectives on rigor and reproducibility in single cell genomics. PLoS Genetics 18(5):e1010210.
von Kortzfleisch, V.T., A. Ambree, N.A. Karp, N. Meyer, J. Novak et al. 2022. Do multiple experimenters improve the reproducibility of animal studies? PLoS Biology 20(5):e3001564.
Wagenmakers, E-J., A. Sarafoglou, and B. Aczel. 2022. One statistical analysis must not rule them all. Nature 605:423-425.
May 2022
Chambers, C.D. and L. Tzavella. 2022. The past, present and future of Registered Reports. Nature Human Behaviour 6:29–42.
Vazire, S., S.R. Schiavone, and J.G. Bottesini. 2022. Credibility beyond replicability: Improving the four validities in psychological science. Current Directions in Psychological Science 31(2):162–168.
April 2022
Debrouwere, S. and Y. Rosseel. 2022. The conceptual, cunning, and conclusive experiment in psychology. Perspectives on Psychological Science 17(3):852–862.
Ross-Hellauer, T. 2022. Open science, done wrong, will compound inequities. Nature 603:363.
March 2022
Marek, S., B. Tervo-Clemmens, F.J. Calabro, et al. 2022. Reproducible brain-wide association studies require thousands of individuals. Nature 603:654–660.
Suls, J., A.J. Rothman, and K.W. Davidson. 2021. Now is the time to assess the effects of open science practices with randomized control trials. American Psychologist.
February 2022
Gross, K. and C.T. Bergstrom. 2021. Why ex post peer review encourages high-risk research while ex ante review discourages it. Proceedings of the National Academy of Sciences 118(51):e2111615118.
Irvine, E. 2021. The role of replication studies in theory building. Perspectives on Psychological Science 16(4):844–853.
January 2022
Hardwicke, T.E., D. Szűcs, R.T. Thibault, S. Crüwell, O. Van den Akker, M.B. Nuijten, and J. Ioannidis. 2021. Citation patterns following a strongly contradictory replication result: Four case studies from psychology. Advances in Methods and Practices in Psychological Science.
McDiarmid, A.D., A.M. Tullett, C.M. Whitt, S. Vazire, P.E. Smaldino, and J.E. Stephens. 2021. Psychologists update their beliefs about effect sizes after replication studies. Nature Human Behaviour 5:1663-1673.
December 2021
Errington, T.M., M.B., Mathur, C.K. Soderberg, A. Denis, N. Perfito, E. Iorns, B.A. Nosek. 2021. Investigating the replicability of preclinical cancer biology. eLife 10:e71601.
Kane, P. and J. Kimmelman. 2021. Is preclinical research in cancer biology reproducible enough? eLife 10:e67527.
Nelson, N.C. 2021. Understand the real reasons reproducibility reform fails. Nature 600:191.
November 2021
Reinagel, P. 2019. Is N-hacking ever OK? A simulation-based study. bioRxiv.
Serghiou, S, D.G. Contopoulos-Ioannidis, K.W. Boyack, N. Riedel, J.D. Wallach , and J.P.A. Ioannidis. 2021. Assessment of transparency indicators across the biomedical literature: How open is open? PLoS Biology 19(3):e3001107.
October 2021
Erayil, S.E., M.K. Smith, T. Gebreslasse, P.F. Walker, E.M. Mann, S. Wilkins, and W.M. Stauffer. 2021. The value and interpretation of race and ethnicity data in the era of global migration: a change is in order. The American Journal of Tropical Medicine and Hygiene.
Nosek, B.A., T.E. Hardwicke, H. Moshontz, A. Allard, K.S. Corker, A. Dreber, F. Fidler, J. Hilgard, M.K. Struhl, M.B. Nuijten, J.M. Rohrer, F. Romero, A.M. Scheel, L.D. Scherer, F.D. Schönbrodt,and S. Vazire. 2022. Replicability, robustness, and reproducibility in psychological science. Annual Review of Psychology 73:1.
Usui, T., M.R. Macleod, S.K. McCann, A.M. Senior, and S. Nakagawa. 2021. Meta-analysis of variation suggests that embracing variability improves both replicability and generalizability in preclinical research. PLoS Biology 19(5):e3001009.
September 2021
Amaral, O.B. and K. Neves. 2021. Reproducibility: expect less of the scientific paper. Nature 597:329–331.
Vachon, B., J.A. Curran, S. Karunananthan, J. Brehaut, I.D. Graham, D. Moher, A.E. Sales, S.E. Straus, M. Fiander, P.A. Paprica, and J.M. Grimshaw. 2021. Changing research culture toward more use of replication research: a narrative review of barriers and strategies. Journal of Clinical Epidemiology 129:21–30.
August 2021
Devezer, B., D.J. Navarro, J. Vandekerckhove, and E. Ozge Buzbas. 2021. The case for formal methodology in scientific reform. Royal Society Open Science 8: 200805.
Soderberg, C.K., T.M. Errington, S.R. Schiavone, J. Bottesini, F. Singleton Thorn, S. Vazire 2,3, K.M. Esterling, and B.A. Nosek. 2021. Initial evidence of research quality of registered reports compared with the standard publishing model. Nature Human Behaviour 5:990–997.
July 2021
Buck, S. 2021. Beware performative reproducibility. Nature 595:151
Peterson, D. 2021. The replication crisis won’t be solved with broad brushstrokes. Nature 594:151
June 2021
Colling, L.J. and D. Szűcs. 2021. Statistical inference and the replication crisis. Review of Philosophy and Psychology 12(1):121-147.
Voelkl, B. and H. Würbel. 2021. A reaction norm perspective on reproducibility. Theory in Biosciences 140:169-176.
May 2021
Peng, R.D. and S.C. Hicks. 2021. Reproducible research: a retrospective. Annual Review of Public Health 42:79-93.
Piller, C. 2021. In a first, FDA cites violation of clinical trials reporting law. Science.
Vinkers, C.H., H.J. Lamberink, J.K. Tijdink, P. Heus, L. Bouter, P. Glasziou et al. 2021. The methodological quality of 176,620 randomized controlled trials published between 1966 and 2018 reveals a positive trend but also an urgent need for improvement. PLoS Biology 19(4):e3001162.
April 2021
Hönekopp, J. and A.H. Linden. 2021. Heterogeneity of research results: a new perspective from which to assess and promote progress in psychological science. Perspectives on Psychological Science 16(2):358–376.
Serghiou, S., D.G. Contopoulos-Ioannidis, K.W. Boyack, N. Riedel, J.D. Wallach, and J.P.A Ioannidis. 2021. Assessment of transparency indicators across the biomedical literature: How open is open? PLoS Biology 19(3):e3001107.
March 2021
Finkel, E.J., P.W. Eastwick, and H.T. Reis. 2017. Replicability and other features of a high-quality science: Toward a balanced and empirical approach. Journal of Personality and Social Psychology 113(2):244-253.
Nichols, J.D., M K. Oli, W.L. Kendall, and G.S. Boomer. 2021. A better approach for dealing with reproducibility and replicability in science. Proceedings of the National Academy of Sciences 118(7):e2100769118.
February 2021
de Haas, B. 2021. What my retraction taught me. Nature 589:331.
Flis, I. 2019. Psychologists psychologizing scientific psychology: An epistemological reading of the replication crisis. Theory & Psychology 29(2):158–181.
Schwarzkopf, S. 2020. When the hole changes the pigeon (blog post related to de Haas 2021).
January 2021
Bakker M., C.L.S Veldkamp, M.A.L.M van Assen, E.A.V Crompvoets, H.H. Ong, B.A. Nosek, et al. 2020. Ensuring the quality and specificity of preregistrations. PLoS Biology 18(12):e3000937.
Romero, F. 2020. The division of replication labor. Philosophy of Science 87:1014-1025.
December 2020
Baker, Z.G., E.-A. Gentzis, E.M. Watlington, S. Castejon, W.E. Petit, M. Britton, S. Haddad, A.M. DiBello, L.M. Rodriguez, J.L. Derrick and C.R. Knee. 2020. Reflections on a registered report replicating a body of dyadic cross-sectional research. Personal Relationships.
Simonsohn, U., J.P. Simmons, and L.D. Nelson. 2020. Specification curve analysis. Nature Human Behaviour 4:1208–1214.
November 2020
da Silva Frost, A. and A. Ledgerwood A. 2020. Calibrate your confidence in research findings: A tutorial on improving research methods and practices. Journal of Pacific Rim Psychology 14:e14.
Mejlgaard, N., L.M. Bouter, G. Gaskell, P. Kavouras, N. Allum, et al. 2020. Research integrity: nine ways to move from talk to walk. Nature 586:358-360.
October 2020
Belikov, A.V. A. Rzhetsky, and J. Evans. 2020. Detecting signal from science: The structure of research communities and prior knowledge improves prediction of genetic regulatory experiments. arXiv:2008.09985
de Menard, A. 2020. What's Wrong with Social Science and How to Fix It: Reflections After Reading 2578 Papers. Fantastic Anachronism
September 2020
Gordon, M., D. Viganola, M. Bishop, Y. Chen, A. Dreber, B. Goldfedder, F. Holzmeister, M. Johannesson, Y. Liu, C. Twardy, J. Wang, and T. Pfeiffer. 2020. Are replication rates the same across academic fields? Community forecasts from the DARPA SCORE programme. Royal Society Open Science 7(7):200566.
Heesen, R. and L.K. Bright. 2020. Is peer review a good idea? The British Journal for the Philosophy of Science: axz029.
August 2020
Jin, J., N. Agarwala, P. Kundu, Y. Wang, R. Zhao, and N. Chatterjee. 2020. Transparency, reproducibility, and validation of COVID-19 projection models. COVID-19: School of Public Health Expert Insights. Johns Hopkins Bloomberg School of Public Health.
Moher, D. L. Bouter, S. Kleinert, P. Glasziou, M.H. Sham, V. Barbour, A.-M. Coriat, N. Foeger, and Ul Dirnagl. 2020. The Hong Kong Principles for assessing researchers: Fostering research integrity. PLoS Biology 18(7):e3000737.
July 2020
Adda, J., C. Decker, and M. Ottaviani. 2020. P-hacking in clinical trials and how incentives shape the distribution of results across phases. Proceedings of the National Academy of Sciences 117:13386-13392.
Botvinik-Nezer, R., F. Holzmeister, C. F. Camerer, A. Dreber, J. Huber, et al. 2020. Variability in the analysis of a single neuroimaging dataset by many teams. Nature 582:84-88.
Ofosu, G.K., and D.N. Posner. 2020. Do pre-analysis plans hamper publication? AEA Papers and Proceedings 110:70-74.
June 2020
May 2020
Gilmore, R.O., P.M, Cole, S. Verma, M.A. Van Aken, and C.M. Worthman. 2020. Advancing scientific integrity, transparency, and openness in child development research: challenges and possible solutions. Child Development Perspectives 14:9-14.
London, A.J. and J. Kimmelman. 2020. Against pandemic research exceptionalism. Science 368:476-477.
Wilkinson, M., M. Dumontier, I. Aalbersberg, et al. 2016. The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data 3:160018.
April 2020
Nosek, B.A. and T.M. Errington. 2020. What is replication? PLoS Biology 18(3): e3000691.
Tiokhin, L., J. Hackman, S. Munira, K. Jesmin, and D. Hruschka. 2019. Generalizability is not optional: insights from a cross-cultural study of social discounting. Royal Society Open Science 6: 181386.
Wilson, B.M., C.R. Harris, J.T. Wixted. 2020. Science is not a signal detection problem. Proceedings of the National Academy of Sciences 117(11):5559–5567.
March 2020
Laskowski, K. 2020. What to do when you don’t trust your data anymore.
Lundwall, R.A. 2019. Changing institutional incentives to foster sound scientific practices: One department. Infant Behavior and Development 55:69-76.
February 2020
Koroshetz, W.J., S. Behrman, C.J. Brame, et al. 2020. Research culture: Framework for advancing rigorous research. eLife 9:e55915.
Milton, M.J.T. and A. Possolo. 2019. Trustworthy data underpin reproducible research. Nature Physics 16:117–119.
January 2020
Bryan, C.J., D.S. Yeager, and J.M. O'Brien. 2019. Replicator degrees of freedom allow publication of misleading failures to replicate. Proceedings of the National Academy of Sciences 116:25535–25545.
Feest, U. 2019. Why replication is overrated. Philosophy of Science 86:895–905.
Krummel, M., C. Blish, M. Kuhns, K. Cadwell, A. Oberst, A. Goldrath, K.M. Ansel, J. Chi, R. O'Connell, E.J. Wherry, M. Pepper, and The Future Immunology Consortium. 2019. Universal principled review: a community-driven method to improve peer review. Cell 179:1441–1445