A meta-analytical answer to the crisis of confidence of Psychology

  • Juan Botella Universidad Autónoma de Madrid
  • Juan I. Durán Centro Universitario Cardenal Cisneros (Madrid)
Keywords: meta-analysis, crisis of confidence, effect size, questionable practices


Meta-analysis is a firmly established methodology and an integral part of the process of generating knowledge across the empirical sciences. Meta-analysis has also focused on methodology and has become a dominant critic of methodological shortcomings. We highlight several problematic issues on how we research in psychology: excess of heterogeneity in the results and difficulties for replication, publication bias, suboptimal methodological quality, and questionable practices of the researchers. These and other problems led to a “crisis of confidence” in psychology. We discuss how the meta-analytical perspective and its procedures can help to overcome the crisis. A more cooperative perspective, instead of a competitive one, can shift to consider replication as a more valuable contribution. Knowledge cannot be based in isolated studies. Given the nature of the object of study of psychology the natural unit to generate knowledge must be the estimated distribution of the effect sizes, not the dichotomous decision on statistical significance in specific studies. Some suggestions are offered on how to redirect researchers' research and practices, so that their personal interests and those of science as such are better aligned.


Download data is not yet available.

Author Biography

Juan Botella, Universidad Autónoma de Madrid

Facultad de Psicologia

Universidad Autonoma de Madrid


Arend, I., Colom, R., Botella, J., Contreras, M. J., Rubio, V, & Santacreu, J. (2003). Quantifying cognitive complexity: evidence from a reasoning task. Personality and Individual Differences, 35, 659-669. doi: doi.org/10.1016/S0191-8869(02)00243-X

Baker, M. (2016). Is there a reproducibility crisis? A Nature survey lifts the lid on how researchers view the ‘crisis’ rocking science and what they think will help. Nature, 533(7604), 452-455. doi: 10.1038/533452a

Bakker, M., van Dijk, A. & Wicherts, J. M. (2012). The Rules of the Game Called Psychological Science. Perspectives on Psychological Science, 7, 543-554. doi: 10.1177/1745691612459060

Blázquez, D., Botella, J., & Suero, M. (2017). The debate on the ego-depletion effect: Evidence from meta-analysis with the p-uniform method. Frontiers in Psychology, 8, 197. doi: 10.3389/fpsyg.2017.00197

Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2010). A basic introduction to fixed-effects and random-effects models for meta-analysis. Research Synthesis Methods, 1, 97-111. doi: 10.1002/jrsm.12

Botella, J., & Eriksen, C. W. (1991). Pattern changes in rapid serial visual presentation tasks without strategic shifts. Bulletin of the Psychonomic Society, 29(2), 105-108.

Botella, J., & Gambara, H. (2006). El meta-análisis: una metodología de nues-tro tiempo. Infocop, 29 mayo.

Botella, J., & Sánchez-Meca, J. (2015). Meta-análisis en ciencias sociales y de la salud. Madrid: Síntesis.

Botella, J., Sepúlveda, A. R., Huang, H., & Gambara, H. (2013). A meta-analysis of the diagnostic accuracy of the SCOFF. Spanish Journal of Psy-chology, 16, e92, 1-8. doi:10.1017/sjp.2013.92

Botella, J., Suero, M., & Ximenez, C. (2012). Análisis de datos en psicología I. Ma-drid: Pirámide.

Botella, J., Ximénez, M. C., Revuelta, J., & Suero, M. (2006). Optimization of sample size in controlled experiments: the CLAST rule. Behavior Research Methods, Instruments & Computers, 38(1), 65-76. doi: 10.3758/BF03192751

Botella, J., & Zamora, Á. (2017). El meta-análisis: una metodología para la in-vestigación en educación. Educación XXI, 20(2), 17-38. doi: 10.5944/educXX1.18241

Cooper, H. M., Hedges, L.V., & Valentine, J.C. (Eds.) (2009). The handbook of research synthesis and meta-analysis (2ª ed.). Nueva York: Russell Sage Founda-tion.

Cumming, G. (2012). Understanding the new statistics: Effect sizes, confidence inter-vals, and meta-analysis. New York, NY: Routledge.

Cumming, G. (2014). The new statistics: why and how. Psychological Science, 25, 7–29. doi: 10.1177/0956797613504966

DeCoster, J., Sparks, E. A., Sparks, J. C., Sparks, G. G., & Sparks, C. W. (2015). Opportunistic biases: Their origins, effects, and an integrated so-lution. American Psychologist, 70(6), 499. doi: 10.1037/a0039191

Earp, B. D., & Trafimow, D. (2015). Replication, falsification, and the crisis of confidence in social psychology. Frontiers in psychology, 6, 621. doi: 10.3389/fpsyg.2015.00621

Frick, R. W. (1996). The appropriate use of null hypothesis testing. Psychological Methods, 1, 379-390. doi: 10.1037/1082-989X.1.4.379

Gigerenzer, G. (1993). The superego, the ego, and the statistical reasoning. En G. Keren y C. Lewis (eds). A Handbook for data analysis in the behavioral sci-ences: methodological issues. Hillsdale, NJ: Erlbaum.

Giner-Sorolla, R. (2012). Science or Art? How Aesthetic Standards Grease the Way Through the Publication Bottleneck but Undermine Science. Perspec-tives on Psychological Science, 7, 562-571. doi: 10.1177/1745691612457576

Glass, G. V. (1976). Primary, secondary and meta-analysis of research. Educa-tional Researcher, 5, 3–8. doi: 10.3102/0013189X005010003

Gøtzsche, P. C., Hróbjartsson, A., Marić, K., & Tendal, B. (2007). Data ex-traction errors in meta-analyses that use standardized mean differences. JAMA, 298(4), 430-437. doi:10.1001/jama.298.4.430

Hagger, M. S., & Chatzisarantis, N. L. D. (2016). A multilab preregistered rep-lication of the ego-depletion effect. Perspectives on Psychological Science, 11(4), 546-573. doi: 10.1177/1745691616652873

Harlow, L. L., Mulaik, S. A., & Steiger, J. H. (1997). What if there were no signifi-cance tests? (pp. 37-64). Mahwah, NJ: Lawrence Erlbaum Associates.

Hattie, J. A. C. (2009). Visible learning: A synthesis of 800+ meta-analyses on achievement. London: Routledge.

Hollenbeck, J. R., & Wright, P.M. (2017). Harking, Sharking, and Tharking: Making the case for post hoc analysis of scientific data. Journal of Manage-ment, 43(1), 5-18. doi: 10.1177/0149206316679487

Ioannidis, J. P. (2005). Why most published research findings are false. PLoS medicine, 2(8), e124. doi: 10.1371/journal.pmed.0020124

Ioannidis, J. (2016). The mass production of redundant, misleading, and con-flicted systematic reviews and meta‐analyses. The Milbank Quarterly, 94(3), 485-514. doi: 10.1111/1468-0009.12210

Ioannidis, J. P., & Trikalinos, T. A. (2007). An exploratory test for an excess of significant findings. Clinical trials, 4(3), 245-253. doi: 10.1177/1740774507079441

John, L., Loewenstein, G. F., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth-telling. Psychologi-cal Science, 23, 524–32. doi: 10.1177/0956797611430953

Klein, O., Doyen, S., Leys, C., Magalhães de Saldanha da Gama, P. A., Miller, S., Questienne, L. & Axel Cleeremans, A. (2012). Low Hopes, High Ex-pectations: Expectancy Effects and the Replicability of Behavioral Exper-iments. Perspectives on Psychological Science, 7, 572-584. doi: 10.1177/1745691612463704

Koole, S. L., & Lakens, D. (2012). Rewarding replications: A sure and simple way to improve psychological science. Perspectives on Psychological Science, 7(6), 608-614. doi: 10.1177/1745691612462586

Lakens, D., Hilgard, J., & Staaks, J. (2016). On the reproducibility of meta-analyses: Six practical recommendations. BMC psychology, 4(1), 24. doi: 10.1186/s40359-016-0126-3

Lash, T. L. (2017). The harm done to reproducibility by the culture of null hy-pothesis significance testing. American Journal of Epidemiology, 186(6), 627-635. doi: 10.1093/aje/kwx261

Lipsey, M. W., & Wilson, D. B. (1993). The efficacy of psychological, educa-tional, and behavioral treatment: Confirmation from meta-analysis. Amer-ican psychologist, 48(12), 1181-1209. doi: 10.1037/0003-066X.48.12.1181

Makel, M. C., Plucker, J. A., & Hegarty, B. (2012). Replications in Psychology Research: How Often Do They Really Occur? Perspectives on Psychological Science, 7, 537-542. doi: 10.1177/1745691612460688

Nickerson, R. S. (2000). Null hypothesis significance testing: a review of an old and continuing controversy. Psychological Methods, 5, 241-301. doi: 10.1037/1082-989X.5.2.241

Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia: II. Restructur-ing incentives and practices to promote truth over publishability. Perspec-tives on Psychological Science, 7, 615–631. doi: 10.1177/1745691612459058

Open Science Collaboration. (2015). Estimating the reproducibility of psycho-logical science. Science, 349(6251), aac4716. doi: 10.1126/science.aac4716

Pashler, H., & Harris, C. R. (2012). Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science, 7(6), 531-536. doi: 10.1177/1745691612463401

Pashler, H., & Wagenmakers, E.J. (2012). Editors’ Introduction to the Special Section on Replicability in Psychological Science: A Crisis of Confidence? Perspectives on Psychological Science, 7, 528-530. doi: 10.1177/1745691612465253

Quiles-Marcos, Y., Quiles-Sebastián, M., Pamies-Aubalat, L., Botella, J., & Treasure, J. (2013). Peer and family influence in eating disorders: A meta-analysis. European Psychiatry, 28(4), 199-206. doi:10.1016/j.eurpsy.2012.03.005

Rosenthal, R. 1979. The “file drawer problem” and tolerance for null results. Psychological Bulletin, 86, 638–641. doi: 10.1037/0033-2909.86.3.638

Rothstein, H. R., Sutton, A. J., & Borenstein, M. (Eds.) (2005). Publication bias in meta-analysis: Prevention, assessment, and adjustments. Nueva York: Wiley.

Schmidt, F. L., & Oh, I. S. (2016). The crisis of confidence in research find-ings in psychology: Is lack of replication the real problem? Or is it some-thing else? Archives of Scientific Psychology, 4(1), 32-37. doi: 10.1037/arc0000029

Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995). Publication deci-sions revisited: The effect of the outcome of statistical tests on the deci-sion to publish and vice versa. American Statistician, 49, 108–112. doi: 10.2307/2684823

Tryon, W. W. (2001). Evaluating statistical difference, equivalence, and inde-terminacy using inferential confidence intervals; an integrated alternative method of conducting null hypothesis statistical tests. Psychological Meth-ods, 6, 371-386. doi: 10.1037/1082-989X.6.4.371

Yong, E. (2012). In the wake of high-profile controversies, psychologists are facing up to problems with replication. Nature, 485, 298-300. doi: 10.1038/485298a
How to Cite
Botella, J., & Durán, J. I. (2019). A meta-analytical answer to the crisis of confidence of Psychology. Anales De Psicología / Annals of Psychology, 35(2), 350-356. https://doi.org/10.6018/analesps.35.2.345291

Most read articles by the same author(s)