Week 4: Science and Publication Bias

Instructions

Some studies have been conducted, but they have never been published. This week, we discuss the causes and consequences of publication bias and what it means for evidence-informed public policy making. Can systematic reviews, evidence maps, and/or meta-analysis help to navigate a biased body of evidence? What are their limits?

Required readings

  • West, J. D. and C. T. Bergstrom. “Misinformation in and about Science”. In: Proceedings of the National Academy of Sciences 118.15 (2021), p. e1912444117. DOI: 10.1073/pnas.1912444117.
  • DellaVigna, S. and E. Linos. “RCTs to Scale: Comprehensive Evidence From Two Nudge Units”. In: Econometrica 90.1 (2022), pp. 81-116. DOI: https://doi.org/10.3982/ECTA18709. (skim only)

Also take a look at two “What works?” websites: Homelessness Effectiveness MapCrime Solutions. If you are curious, the Resource page lists more.

Further reading

  • Frey, B. S. “Publishing as Prostitution?-Choosing Between One’s Own Ideas and Academic Success”. In: Public choice 116.1 (2003), pp. 205-223.
  • Pielke Jr, R. A. The Honest Broker: Making Sense of Science in Policy and Politics. Cambridge: Cambridge University Press, 2007. DOI: 10.1017/CBO9780511818110.
  • Franco, A., N. Malhotra, and G. Simonovits. “Publication Bias in the Social Sciences: Unlocking the File Drawer”. In: Science 345.6203 (2014), pp. 1502-1505.
  • Ioannidis, J. P. A. “Why Most Published Research Findings Are False”. In: PLoS Medicine 2.8 (2005), p. e124.

Another meta-analysis on nudging that received some critical attention in summer 2022:

  • Mertens, S., M. Herberz, U. J. Hahnel, et al. “The Effectiveness of Nudging: A Meta-analysis of Choice Architecture Interventions across Behavioral Domains”. In: Proceedings of the National Academy of Sciences 119.1 (2022), p. e2107346118. DOI: 10.1073/pnas.2107346118.
    • Bakdash, J. Z. and L. R. Marusich. “Left-truncated Effects and Overestimated Meta-analytic Means”. In: Proceedings of the National Academy of Sciences 119.31 (2022), p. e2203616119. DOI: 10.1073/pnas.2203616119.
    • Maier, M., F. Bartoš, T. Stanley, et al. “No Evidence for Nudging After Adjusting for Publication Bias”. In: Proceedings of the National Academy of Sciences 119.31 (2022), p. e2200300119. DOI: 10.1073/pnas.2200300119.
    • Szaszi, B., A. Higney, A. Charlton, et al. “No Reason to Expect Large and Consistent Effects of Nudge Interventions”. In: Proceedings of the National Academy of Sciences 119.31 (2022), p. e2200732119. DOI: 10.1073/pnas.220073211.
    • Mertens, S., M. Herberz, U. J. Hahnel, et al. “Reply to Maier et al., Szaszi et al., and Bakdash and Marusich: The Present and Future of Choice Architecture Research”. In: Proceedings of the National Academy of Sciences 119.31 (2022), p. e2202928119. DOI: 10.1073/pnas.2202928119.

Suggested media

Acknowledgments

Photo credit: https://unsplash.com/photos/4T7-GLBDLKE.