The Cutting Edge - What We're Reading

Narrow banner

A lot of new interesting work is being done on questions of reproducibility across the sciences by diverse researchers from different disciplines. Over the course of the project we shared 2-3 papers every month that come from what Project Members are reading. No further entries will be made now that this project has concluded (June 2023).

Expand all

May 2023

Hostler, T.J. 2023. The invisible workload of open research. Journal of Trial & Error.

Spirling, A. 2023. Why open-source generative AI models are an ethical way forward for science. Nature 616:413.

University of Vermont's Faculty Senate Resolution on Open Access and Open Science

April 2023

Ciubotariu, I.I. and G. Bosch. 2022. Improving research integrity: a framework for responsible science communication. BMC Research Notes 15:177.

Flake, J.K., I.J. Davidson, O. Wong, and J. Pek. 2022. Construct validity and the validity of replication studies: A systematic review. American Psychologist 77:576–588.

March 2023

Cobey, K.D., S. Haustein, J. Brehaut, U. Dirnagl, D.L. Franzen, L.G. Hemkens, et al. 2023. Community consensus on core open science practices to monitor in biomedicine. PLoS Biology 21(1):e3001949.

Ghai, S., L. Fassi, F. Awadh,and A. Orben. 2023. Lack of sample diversity in research on adolescent depression and social media use: A scoping review and meta-analysis. Clinical Psychological Science.

Kummerfeld, E. and G.L. Jones. 2023. One data set, many analysts: Implications for practicing scientists. Frontiers in Psychology 14:1094150.

February 2023

Crüwell, S., D. Apthorp, B.J. Baker, L. Colling, M. Elson et al. 2023. What’s in a badge? A computational reproducibility investigation of the open data badge policy in one issue of Psychological Science. Psychological Science.

Ross-Hellauer, T. 2023. Strategic priorities for reproducibility reform. PLoS Biology 21(1):e3001943.

Youyou, W., Y. Yang, and B. Uzzi. 2023. A discipline-wide investigation of the replicability of psychology papers over the past two decades. Proceedings of the National Academy of Sciences 120(6):e2208863120.

January 2023

Anderegg, W.R.L., A.T. Trugman, J. Wang, and C. Wu. 2022. Open science priorities for rigorous nature-based climate solutions. PLoS Biology 20(12):e3001929.

Coles, N.A., D.S. March, F. Marmolejo-Ramos, et al. 2022. A multi-lab test of the facial feedback hypothesis by the Many Smiles Collaboration. Nature Human Behaviour 6:1731–1742.

December 2022

Arroyo-Araujo, M., B. Voelkl, C. Laloux, J. Novak, B. Koopmans, et al. 2022. Systematic assessment of the replicability and generalizability of preclinical findings: Impact of protocol harmonization across laboratory sites. PLoS Biology 20(11): e3001886.

Colaço, D. J. Bickle, and B Walters. 2022. When should researchers cite study differences in response to a failure to replicate? Biology & Philosophy 37:39.

November 2022

Bik, E. 2022. Science has a nasty photoshopping problem. New York Times (November 2nd).

Breznau, N., E.M. Rinke, A. Wuttke, H.H.V. Nguyen, M. Ademe, et al. 2022. Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty. Proceedings of the National Academy of Sciences 119:e2203150119.

October 2022

Kapoor, S. and A. Narayanan. 2022. Leakage and the reproducibility crisis in ML-based science. arXiv:2207.07048

Sabatello, M. D.O. Martschenko, M.K. Cho, and K.B. Brothers. 2022. Data sharing and community-engaged research. Science 378:141-143.

September 2022

Horbach, S.P.J.M., L.M. Bouter, G. Gaskell, M. Hiney, P. Kavouras, et al. 2022. Designing and implementing a research integrity promotion plan: Recommendations for research funders. PLoS Biology 20(8):e3001773.

Musen, M.A. 2022. Without appropriate metadata, data-sharing mandates are pointless. Nature 609:222. doi:

August 2022

Delios, A., E.G. Clemente, T. Wu, H. Tan, Y. Wang, et al. 2022. Examining the generalizability of research findings from archival data. Proceedings of the National Academy of Sciences 119(3)):e2120377119.

Kent, B.A., C. Holman, E. Amoako, A. Antonietti, J.M. Azam, et al. 2022. Recommendations for empowering early career researchers to improve research culture and practice. PLoS Biology 20(7):e3001680.

June 2022

Gibson, G. 2022. Perspectives on rigor and reproducibility in single cell genomics. PLoS Genetics 18(5):e1010210.

von Kortzfleisch, V.T., A. Ambree, N.A. Karp, N. Meyer, J. Novak et al. 2022. Do multiple experimenters improve the reproducibility of animal studies? PLoS Biology 20(5):e3001564.

Wagenmakers, E-J., A. Sarafoglou, and B. Aczel. 2022. One statistical analysis must not rule them all. Nature 605:423-425.

May 2022

Chambers, C.D. and L. Tzavella. 2022. The past, present and future of Registered Reports. Nature Human Behaviour 6:29–42.

Vazire, S., S.R. Schiavone, and J.G. Bottesini. 2022. Credibility beyond replicability: Improving the four validities in psychological science. Current Directions in Psychological Science 31(2):162–168.

April 2022

Debrouwere, S. and Y. Rosseel. 2022. The conceptual, cunning, and conclusive experiment in psychology. Perspectives on Psychological Science 17(3):852–862.

Ross-Hellauer, T. 2022. Open science, done wrong, will compound inequities. Nature 603:363.

March 2022

Marek, S., B. Tervo-Clemmens, F.J. Calabro, et al. 2022. Reproducible brain-wide association studies require thousands of individuals. Nature 603:654–660.

Suls, J., A.J. Rothman, and K.W. Davidson. 2021. Now is the time to assess the effects of open science practices with randomized control trials. American Psychologist.

February 2022

Gross, K. and C.T. Bergstrom. 2021. Why ex post peer review encourages high-risk research while ex ante review discourages it. Proceedings of the National Academy of Sciences 118(51):e2111615118.

Irvine, E. 2021. The role of replication studies in theory building. Perspectives on Psychological Science 16(4):844–853.

January 2022

Hardwicke, T.E., D. Szűcs, R.T. Thibault, S. Crüwell, O. Van den Akker, M.B. Nuijten, and J. Ioannidis. 2021. Citation patterns following a strongly contradictory replication result: Four case studies from psychology. Advances in Methods and Practices in Psychological Science.

McDiarmid, A.D., A.M. Tullett, C.M. Whitt, S. Vazire, P.E. Smaldino, and J.E. Stephens. 2021. Psychologists update their beliefs about effect sizes after replication studies. Nature Human Behaviour 5:1663-1673.

December 2021

Errington, T.M., M.B., Mathur, C.K. Soderberg, A. Denis, N. Perfito, E. Iorns, B.A. Nosek. 2021. Investigating the replicability of preclinical cancer biology. eLife 10:e71601.

Kane, P. and J. Kimmelman. 2021. Is preclinical research in cancer biology reproducible enough? eLife 10:e67527.

Nelson, N.C. 2021. Understand the real reasons reproducibility reform fails. Nature 600:191.

November 2021

Reinagel, P. 2019. Is N-hacking ever OK? A simulation-based study. bioRxiv.

Serghiou, S, D.G. Contopoulos-Ioannidis, K.W. Boyack, N. Riedel, J.D. Wallach , and J.P.A. Ioannidis. 2021. Assessment of transparency indicators across the biomedical literature: How open is open? PLoS Biology 19(3):e3001107.

October 2021

Erayil, S.E., M.K. Smith, T. Gebreslasse, P.F. Walker, E.M. Mann, S. Wilkins, and W.M. Stauffer. 2021. The value and interpretation of race and ethnicity data in the era of global migration: a change is in order. The American Journal of Tropical Medicine and Hygiene.

Nosek, B.A., T.E. Hardwicke, H. Moshontz, A. Allard, K.S. Corker, A. Dreber, F. Fidler, J. Hilgard, M.K. Struhl, M.B. Nuijten, J.M. Rohrer, F. Romero, A.M. Scheel, L.D. Scherer, F.D. Schönbrodt,and S. Vazire. 2022. Replicability, robustness, and reproducibility in psychological science. Annual Review of Psychology 73:1.

Usui, T., M.R. Macleod, S.K. McCann, A.M. Senior, and S. Nakagawa. 2021. Meta-analysis of variation suggests that embracing variability improves both replicability and generalizability in preclinical research. PLoS Biology 19(5):e3001009.

September 2021

Amaral, O.B. and K. Neves. 2021. Reproducibility: expect less of the scientific paper. Nature 597:329–331.

Vachon, B., J.A. Curran, S. Karunananthan, J. Brehaut, I.D. Graham, D. Moher, A.E. Sales, S.E. Straus, M. Fiander, P.A. Paprica, and J.M. Grimshaw. 2021. Changing research culture toward more use of replication research: a narrative review of barriers and strategies. Journal of Clinical Epidemiology 129:21–30.

August 2021

Devezer, B., D.J. Navarro, J. Vandekerckhove, and E. Ozge Buzbas. 2021. The case for formal methodology in scientific reform. Royal Society Open Science 8: 200805.

Soderberg, C.K., T.M. Errington, S.R. Schiavone, J. Bottesini, F. Singleton Thorn, S. Vazire 2,3, K.M. Esterling, and B.A. Nosek. 2021. Initial evidence of research quality of registered reports compared with the standard publishing model. Nature Human Behaviour 5:990–997.

July 2021

Buck, S. 2021. Beware performative reproducibility. Nature 595:151

Peterson, D. 2021. The replication crisis won’t be solved with broad brushstrokes. Nature 594:151

June 2021

Colling, L.J. and D. Szűcs. 2021. Statistical inference and the replication crisis. Review of Philosophy and Psychology 12(1):121-147.

Voelkl, B. and H. Würbel. 2021. A reaction norm perspective on reproducibility. Theory in Biosciences 140:169-176.

May 2021

Peng, R.D. and S.C. Hicks. 2021. Reproducible research: a retrospective. Annual Review of Public Health 42:79-93.

Piller, C. 2021. In a first, FDA cites violation of clinical trials reporting law. Science.

Vinkers, C.H., H.J. Lamberink, J.K. Tijdink, P. Heus, L. Bouter, P. Glasziou et al. 2021. The methodological quality of 176,620 randomized controlled trials published between 1966 and 2018 reveals a positive trend but also an urgent need for improvement. PLoS Biology 19(4):e3001162.

April 2021

Hönekopp, J. and A.H. Linden. 2021. Heterogeneity of research results: a new perspective from which to assess and promote progress in psychological science. Perspectives on Psychological Science 16(2):358–376.

Serghiou, S., D.G. Contopoulos-Ioannidis, K.W. Boyack, N. Riedel, J.D. Wallach, and J.P.A Ioannidis. 2021. Assessment of transparency indicators across the biomedical literature: How open is open? PLoS Biology 19(3):e3001107.

March 2021

Finkel, E.J., P.W. Eastwick, and H.T. Reis. 2017. Replicability and other features of a high-quality science: Toward a balanced and empirical approach. Journal of Personality and Social Psychology 113(2):244-253.

Nichols, J.D., M K. Oli, W.L. Kendall, and G.S. Boomer. 2021. A better approach for dealing with reproducibility and replicability in science. Proceedings of the National Academy of Sciences 118(7):e2100769118.

February 2021

de Haas, B. 2021. What my retraction taught me. Nature 589:331.

Flis, I. 2019. Psychologists psychologizing scientific psychology: An epistemological reading of the replication crisis. Theory & Psychology 29(2):158–181.

Schwarzkopf, S. 2020. When the hole changes the pigeon (blog post related to de Haas 2021).

January 2021

Bakker M., C.L.S Veldkamp, M.A.L.M van Assen, E.A.V Crompvoets, H.H. Ong, B.A. Nosek, et al. 2020. Ensuring the quality and specificity of preregistrations. PLoS Biology 18(12):e3000937.

Romero, F. 2020. The division of replication labor. Philosophy of Science 87:1014-1025.

December 2020

Baker, Z.G., E.-A. Gentzis, E.M. Watlington, S. Castejon, W.E. Petit, M. Britton, S. Haddad, A.M. DiBello, L.M. Rodriguez, J.L. Derrick and C.R. Knee. 2020. Reflections on a registered report replicating a body of dyadic cross-sectional research. Personal Relationships.

Simonsohn, U., J.P. Simmons, and L.D. Nelson. 2020. Specification curve analysis. Nature Human Behaviour 4:1208–1214.

November 2020

da Silva Frost, A. and A. Ledgerwood A. 2020. Calibrate your confidence in research findings: A tutorial on improving research methods and practices. Journal of Pacific Rim Psychology 14:e14.

Mejlgaard, N., L.M. Bouter, G. Gaskell, P. Kavouras, N. Allum, et al. 2020. Research integrity: nine ways to move from talk to walk. Nature 586:358-360.

September 2020

Gordon, M., D. Viganola, M. Bishop, Y. Chen, A. Dreber, B. Goldfedder, F. Holzmeister, M. Johannesson, Y. Liu, C. Twardy, J. Wang, and T. Pfeiffer. 2020. Are replication rates the same across academic fields? Community forecasts from the DARPA SCORE programme. Royal Society Open Science 7(7):200566.

Heesen, R. and L.K. Bright. 2020. Is peer review a good idea? The British Journal for the Philosophy of Science: axz029.

August 2020

Jin, J., N. Agarwala, P. Kundu, Y. Wang, R. Zhao, and N. Chatterjee. 2020. Transparency, reproducibility, and validation of COVID-19 projection models. COVID-19: School of Public Health Expert Insights. Johns Hopkins Bloomberg School of Public Health.

Moher, D. L. Bouter, S. Kleinert, P. Glasziou, M.H. Sham, V. Barbour, A.-M. Coriat, N. Foeger, and Ul Dirnagl. 2020. The Hong Kong Principles for assessing researchers: Fostering research integrity. PLoS Biology 18(7):e3000737.

July 2020

Adda, J., C. Decker, and M. Ottaviani. 2020. P-hacking in clinical trials and how incentives shape the distribution of results across phases. Proceedings of the National Academy of Sciences 117:13386-13392.

Botvinik-Nezer, R., F. Holzmeister, C. F. Camerer, A. Dreber, J. Huber, et al. 2020. Variability in the analysis of a single neuroimaging dataset by many teams. Nature 582:84-88.

Ofosu, G.K., and D.N. Posner. 2020. Do pre-analysis plans hamper publication? AEA Papers and Proceedings 110:70-74.

June 2020

Cowan, N., C. Belletier, J.M. Doherty, A.J. Jaroslawska, S. Rhodes, A. Forsberg, M. Naveh-Benjamin, P. Barrouillet, V. Camos, and R.H. Logie. 2020. How do scientific views change? Notes from an extended adversarial collaboration. Perspectives on Psychological Science.
Poldrack, R.A. 2019. The costs of reproducibility. Neuron 101:11-14.

May 2020

Gilmore, R.O., P.M, Cole, S. Verma, M.A. Van Aken, and C.M. Worthman. 2020. Advancing scientific integrity, transparency, and openness in child development research: challenges and possible solutions. Child Development Perspectives 14:9-14.

London, A.J. and J. Kimmelman. 2020. Against pandemic research exceptionalism. Science 368:476-477.

Wilkinson, M., M. Dumontier, I. Aalbersberg, et al. 2016. The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data 3:160018.

April 2020

Nosek, B.A. and T.M. Errington. 2020. What is replication? PLoS Biology 18(3): e3000691.

Tiokhin, L., J. Hackman, S. Munira, K. Jesmin, and D. Hruschka. 2019. Generalizability is not optional: insights from a cross-cultural study of social discounting. Royal Society Open Science 6: 181386.

Wilson, B.M., C.R. Harris, J.T. Wixted. 2020. Science is not a signal detection problem. Proceedings of the National Academy of Sciences 117(11):5559–5567.

March 2020

February 2020

Koroshetz, W.J., S. Behrman, C.J. Brame, et al. 2020. Research culture: Framework for advancing rigorous research. eLife 9:e55915.

Milton, M.J.T. and A. Possolo. 2019. Trustworthy data underpin reproducible research. Nature Physics 16:117–119.

January 2020

Bryan, C.J., D.S. Yeager, and J.M. O'Brien. 2019. Replicator degrees of freedom allow publication of misleading failures to replicate. Proceedings of the National Academy of Sciences 116:25535–25545.

Feest, U. 2019. Why replication is overrated. Philosophy of Science 86:895–905.

Krummel, M., C. Blish, M. Kuhns, K. Cadwell, A. Oberst, A. Goldrath, K.M. Ansel, J. Chi, R. O'Connell, E.J. Wherry, M. Pepper, and The Future Immunology Consortium. 2019. Universal principled review: a community-driven method to improve peer review. Cell 179:1441–1445