The Cutting Edge - What We're Reading

Crüwell, S., D. Apthorp, B.J. Baker, L. Colling, M. Elson et al. 2023. What’s in a badge? A computational reproducibility investigation of the open data badge policy in one issue of Psychological Science. Psychological Science.

Ross-Hellauer, T. 2023. Strategic priorities for reproducibility reform. PLoS Biology 21(1):e3001943.

Youyou, W., Y. Yang, and B. Uzzi. 2023. A discipline-wide investigation of the replicability of psychology papers over the past two decades. Proceedings of the National Academy of Sciences 120(6):e2208863120.

Anderegg, W.R.L., A.T. Trugman, J. Wang, and C. Wu. 2022. Open science priorities for rigorous nature-based climate solutions. PLoS Biology 20(12):e3001929.

Coles, N.A., D.S. March, F. Marmolejo-Ramos, et al. 2022. A multi-lab test of the facial feedback hypothesis by the Many Smiles Collaboration. Nature Human Behaviour 6:1731–1742.

Arroyo-Araujo, M., B. Voelkl, C. Laloux, J. Novak, B. Koopmans, et al. 2022. Systematic assessment of the replicability and generalizability of preclinical findings: Impact of protocol harmonization across laboratory sites. PLoS Biology 20(11): e3001886.

Colaço, D. J. Bickle, and B Walters. 2022. When should researchers cite study differences in response to a failure to replicate? Biology & Philosophy 37:39.

Bik, E. 2022. Science has a nasty photoshopping problem. New York Times (November 2nd).

Breznau, N., E.M. Rinke, A. Wuttke, H.H.V. Nguyen, M. Ademe, et al. 2022. Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty. Proceedings of the National Academy of Sciences 119:e2203150119.

Kapoor, S. and A. Narayanan. 2022. Leakage and the reproducibility crisis in ML-based science. arXiv:2207.07048

Sabatello, M. D.O. Martschenko, M.K. Cho, and K.B. Brothers. 2022. Data sharing and community-engaged research. Science 378:141-143.

Horbach, S.P.J.M., L.M. Bouter, G. Gaskell, M. Hiney, P. Kavouras, et al. 2022. Designing and implementing a research integrity promotion plan: Recommendations for research funders. PLoS Biology 20(8):e3001773.

Musen, M.A. 2022. Without appropriate metadata, data-sharing mandates are pointless. Nature 609:222. doi: https://doi.org/10.1038/d41586-022-02820-7

Delios, A., E.G. Clemente, T. Wu, H. Tan, Y. Wang, et al. 2022. Examining the generalizability of research findings from archival data. Proceedings of the National Academy of Sciences 119(3)):e2120377119.

Kent, B.A., C. Holman, E. Amoako, A. Antonietti, J.M. Azam, et al. 2022. Recommendations for empowering early career researchers to improve research culture and practice. PLoS Biology 20(7):e3001680.

Mayo, D.G. and D. Hand. 2022. Statistical significance and its critics: practicing damaging science, or damaging scientific practice? Synthese 200:220.

Stewart, S.L.K., Pennington, C.R., da Silva, G.R. et al. 2022. Reforms to improve reproducibility and quality must be coordinated across the research ecosystem: the view from the UKRN Local Network Leads. BMC Research Notes 15:58.

Gibson, G. 2022. Perspectives on rigor and reproducibility in single cell genomics. PLoS Genetics 18(5):e1010210.

von Kortzfleisch, V.T., A. Ambree, N.A. Karp, N. Meyer, J. Novak et al. 2022. Do multiple experimenters improve the reproducibility of animal studies? PLoS Biology 20(5):e3001564.

Wagenmakers, E-J., A. Sarafoglou, and B. Aczel. 2022. One statistical analysis must not rule them all. Nature 605:423-425.

Chambers, C.D. and L. Tzavella. 2022. The past, present and future of Registered Reports. Nature Human Behaviour 6:29–42.

Vazire, S., S.R. Schiavone, and J.G. Bottesini. 2022. Credibility beyond replicability: Improving the four validities in psychological science. Current Directions in Psychological Science 31(2):162–168.

Debrouwere, S. and Y. Rosseel. 2022. The conceptual, cunning, and conclusive experiment in psychology. Perspectives on Psychological Science 17(3):852–862.

Ross-Hellauer, T. 2022. Open science, done wrong, will compound inequities. Nature 603:363.

Marek, S., B. Tervo-Clemmens, F.J. Calabro, et al. 2022. Reproducible brain-wide association studies require thousands of individuals. Nature 603:654–660.

Suls, J., A.J. Rothman, and K.W. Davidson. 2021. Now is the time to assess the effects of open science practices with randomized control trials. American Psychologist.

Gross, K. and C.T. Bergstrom. 2021. Why ex post peer review encourages high-risk research while ex ante review discourages it. Proceedings of the National Academy of Sciences 118(51):e2111615118.

Irvine, E. 2021. The role of replication studies in theory building. Perspectives on Psychological Science 16(4):844–853.

Hardwicke, T.E., D. Szűcs, R.T. Thibault, S. Crüwell, O. Van den Akker, M.B. Nuijten, and J. Ioannidis. 2021. Citation patterns following a strongly contradictory replication result: Four case studies from psychology. Advances in Methods and Practices in Psychological Science.

McDiarmid, A.D., A.M. Tullett, C.M. Whitt, S. Vazire, P.E. Smaldino, and J.E. Stephens. 2021. Psychologists update their beliefs about effect sizes after replication studies. Nature Human Behaviour 5:1663-1673.

Errington, T.M., M.B., Mathur, C.K. Soderberg, A. Denis, N. Perfito, E. Iorns, B.A. Nosek. 2021. Investigating the replicability of preclinical cancer biology. eLife 10:e71601.

Kane, P. and J. Kimmelman. 2021. Is preclinical research in cancer biology reproducible enough? eLife 10:e67527.

Nelson, N.C. 2021. Understand the real reasons reproducibility reform fails. Nature 600:191.

Reinagel, P. 2019. Is N-hacking ever OK? A simulation-based study. bioRxiv.

Serghiou, S, D.G. Contopoulos-Ioannidis, K.W. Boyack, N. Riedel, J.D. Wallach , and J.P.A. Ioannidis. 2021. Assessment of transparency indicators across the biomedical literature: How open is open? PLoS Biology 19(3):e3001107.

Erayil, S.E., M.K. Smith, T. Gebreslasse, P.F. Walker, E.M. Mann, S. Wilkins, and W.M. Stauffer. 2021. The value and interpretation of race and ethnicity data in the era of global migration: a change is in order. The American Journal of Tropical Medicine and Hygiene.

Nosek, B.A., T.E. Hardwicke, H. Moshontz, A. Allard, K.S. Corker, A. Dreber, F. Fidler, J. Hilgard, M.K. Struhl, M.B. Nuijten, J.M. Rohrer, F. Romero, A.M. Scheel, L.D. Scherer, F.D. Schönbrodt,and S. Vazire. 2022. Replicability, robustness, and reproducibility in psychological science. Annual Review of Psychology 73:1.

Usui, T., M.R. Macleod, S.K. McCann, A.M. Senior, and S. Nakagawa. 2021. Meta-analysis of variation suggests that embracing variability improves both replicability and generalizability in preclinical research. PLoS Biology 19(5):e3001009.

Amaral, O.B. and K. Neves. 2021. Reproducibility: expect less of the scientific paper. Nature 597:329–331.

Vachon, B., J.A. Curran, S. Karunananthan, J. Brehaut, I.D. Graham, D. Moher, A.E. Sales, S.E. Straus, M. Fiander, P.A. Paprica, and J.M. Grimshaw. 2021. Changing research culture toward more use of replication research: a narrative review of barriers and strategies. Journal of Clinical Epidemiology 129:21–30.

Devezer, B., D.J. Navarro, J. Vandekerckhove, and E. Ozge Buzbas. 2021. The case for formal methodology in scientific reform. Royal Society Open Science 8: 200805.

Soderberg, C.K., T.M. Errington, S.R. Schiavone, J. Bottesini, F. Singleton Thorn, S. Vazire 2,3, K.M. Esterling, and B.A. Nosek. 2021. Initial evidence of research quality of registered reports compared with the standard publishing model. Nature Human Behaviour 5:990–997.

Buck, S. 2021. Beware performative reproducibility. Nature 595:151

Peterson, D. 2021. The replication crisis won’t be solved with broad brushstrokes. Nature 594:151

Colling, L.J. and D. Szűcs. 2021. Statistical inference and the replication crisis. Review of Philosophy and Psychology 12(1):121-147.

Voelkl, B. and H. Würbel. 2021. A reaction norm perspective on reproducibility. Theory in Biosciences 140:169-176.

Peng, R.D. and S.C. Hicks. 2021. Reproducible research: a retrospective. Annual Review of Public Health 42:79-93.

Piller, C. 2021. In a first, FDA cites violation of clinical trials reporting law. Science.

Vinkers, C.H., H.J. Lamberink, J.K. Tijdink, P. Heus, L. Bouter, P. Glasziou et al. 2021. The methodological quality of 176,620 randomized controlled trials published between 1966 and 2018 reveals a positive trend but also an urgent need for improvement. PLoS Biology 19(4):e3001162.

Hönekopp, J. and A.H. Linden. 2021. Heterogeneity of research results: a new perspective from which to assess and promote progress in psychological science. Perspectives on Psychological Science 16(2):358–376.

Serghiou, S., D.G. Contopoulos-Ioannidis, K.W. Boyack, N. Riedel, J.D. Wallach, and J.P.A Ioannidis. 2021. Assessment of transparency indicators across the biomedical literature: How open is open? PLoS Biology 19(3):e3001107.

Finkel, E.J., P.W. Eastwick, and H.T. Reis. 2017. Replicability and other features of a high-quality science: Toward a balanced and empirical approach. Journal of Personality and Social Psychology 113(2):244-253.

Nichols, J.D., M K. Oli, W.L. Kendall, and G.S. Boomer. 2021. A better approach for dealing with reproducibility and replicability in science. Proceedings of the National Academy of Sciences 118(7):e2100769118.

de Haas, B. 2021. What my retraction taught me. Nature 589:331.

Flis, I. 2019. Psychologists psychologizing scientific psychology: An epistemological reading of the replication crisis. Theory & Psychology 29(2):158–181.

Schwarzkopf, S. 2020. When the hole changes the pigeon (blog post related to de Haas 2021).

Bakker M., C.L.S Veldkamp, M.A.L.M van Assen, E.A.V Crompvoets, H.H. Ong, B.A. Nosek, et al. 2020. Ensuring the quality and specificity of preregistrations. PLoS Biology 18(12):e3000937.

Romero, F. 2020. The division of replication labor. Philosophy of Science 87:1014-1025.

Baker, Z.G., E.-A. Gentzis, E.M. Watlington, S. Castejon, W.E. Petit, M. Britton, S. Haddad, A.M. DiBello, L.M. Rodriguez, J.L. Derrick and C.R. Knee. 2020. Reflections on a registered report replicating a body of dyadic cross-sectional research. Personal Relationships.

Simonsohn, U., J.P. Simmons, and L.D. Nelson. 2020. Specification curve analysis. Nature Human Behaviour 4:1208–1214.

da Silva Frost, A. and A. Ledgerwood A. 2020. Calibrate your confidence in research findings: A tutorial on improving research methods and practices. Journal of Pacific Rim Psychology 14:e14.

Mejlgaard, N., L.M. Bouter, G. Gaskell, P. Kavouras, N. Allum, et al. 2020. Research integrity: nine ways to move from talk to walk. Nature 586:358-360.

Belikov, A.V. A. Rzhetsky, and J. Evans. 2020. Detecting signal from science: The structure of research communities and prior knowledge improves prediction of genetic regulatory experiments. arXiv:2008.09985

de Menard, A. 2020. What's Wrong with Social Science and How to Fix It: Reflections After Reading 2578 Papers. Fantastic Anachronism

Gordon, M., D. Viganola, M. Bishop, Y. Chen, A. Dreber, B. Goldfedder, F. Holzmeister, M. Johannesson, Y. Liu, C. Twardy, J. Wang, and T. Pfeiffer. 2020. Are replication rates the same across academic fields? Community forecasts from the DARPA SCORE programme. Royal Society Open Science 7(7):200566.

Heesen, R. and L.K. Bright. 2020. Is peer review a good idea? The British Journal for the Philosophy of Science: axz029.

Jin, J., N. Agarwala, P. Kundu, Y. Wang, R. Zhao, and N. Chatterjee. 2020. Transparency, reproducibility, and validation of COVID-19 projection models. COVID-19: School of Public Health Expert Insights. Johns Hopkins Bloomberg School of Public Health.

Moher, D. L. Bouter, S. Kleinert, P. Glasziou, M.H. Sham, V. Barbour, A.-M. Coriat, N. Foeger, and Ul Dirnagl. 2020. The Hong Kong Principles for assessing researchers: Fostering research integrity. PLoS Biology 18(7):e3000737.

Adda, J., C. Decker, and M. Ottaviani. 2020. P-hacking in clinical trials and how incentives shape the distribution of results across phases. Proceedings of the National Academy of Sciences 117:13386-13392.

Botvinik-Nezer, R., F. Holzmeister, C. F. Camerer, A. Dreber, J. Huber, et al. 2020. Variability in the analysis of a single neuroimaging dataset by many teams. Nature 582:84-88.

Ofosu, G.K., and D.N. Posner. 2020. Do pre-analysis plans hamper publication? AEA Papers and Proceedings 110:70-74.

Gilmore, R.O., P.M, Cole, S. Verma, M.A. Van Aken, and C.M. Worthman. 2020. Advancing scientific integrity, transparency, and openness in child development research: challenges and possible solutions. Child Development Perspectives 14:9-14.

London, A.J. and J. Kimmelman. 2020. Against pandemic research exceptionalism. Science 368:476-477.

Wilkinson, M., M. Dumontier, I. Aalbersberg, et al. 2016. The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data 3:160018.

Nosek, B.A. and T.M. Errington. 2020. What is replication? PLoS Biology 18(3): e3000691.

Tiokhin, L., J. Hackman, S. Munira, K. Jesmin, and D. Hruschka. 2019. Generalizability is not optional: insights from a cross-cultural study of social discounting. Royal Society Open Science 6: 181386.

Wilson, B.M., C.R. Harris, J.T. Wixted. 2020. Science is not a signal detection problem. Proceedings of the National Academy of Sciences 117(11):5559–5567.

Laskowski, K. 2020. What to do when you don’t trust your data anymore.

Lundwall, R.A. 2019. Changing institutional incentives to foster sound scientific practices: One department. Infant Behavior and Development 55:69-76.

Koroshetz, W.J., S. Behrman, C.J. Brame, et al. 2020. Research culture: Framework for advancing rigorous research. eLife 9:e55915.

Milton, M.J.T. and A. Possolo. 2019. Trustworthy data underpin reproducible research. Nature Physics 16:117–119.

Bryan, C.J., D.S. Yeager, and J.M. O'Brien. 2019. Replicator degrees of freedom allow publication of misleading failures to replicate. Proceedings of the National Academy of Sciences 116:25535–25545.

Feest, U. 2019. Why replication is overrated. Philosophy of Science 86:895–905.

Krummel, M., C. Blish, M. Kuhns, K. Cadwell, A. Oberst, A. Goldrath, K.M. Ansel, J. Chi, R. O'Connell, E.J. Wherry, M. Pepper, and The Future Immunology Consortium. 2019. Universal principled review: a community-driven method to improve peer review. Cell 179:1441–1445