A meta-analytical answer to the crisis of confidence of Psychology
Abstract
Meta-analysis is a firmly established methodology and an integral part of the process of generating knowledge across the empirical sciences. Meta-analysis has also focused on methodology and has become a dominant critic of methodological shortcomings. We highlight several problematic issues on how we research in psychology: excess of heterogeneity in the results and difficulties for replication, publication bias, suboptimal methodological quality, and questionable practices of the researchers. These and other problems led to a “crisis of confidence” in psychology. We discuss how the meta-analytical perspective and its procedures can help to overcome the crisis. A more cooperative perspective, instead of a competitive one, can shift to consider replication as a more valuable contribution. Knowledge cannot be based in isolated studies. Given the nature of the object of study of psychology the natural unit to generate knowledge must be the estimated distribution of the effect sizes, not the dichotomous decision on statistical significance in specific studies. Some suggestions are offered on how to redirect researchers' research and practices, so that their personal interests and those of science as such are better aligned.
Downloads
References
Arend, I., Colom, R., Botella, J., Contreras, M. J., Rubio, V, & Santacreu, J. (2003). Quantifying cognitive complexity: evidence from a reasoning task. Personality and Individual Differences, 35, 659-669. doi: doi.org/10.1016/S0191-8869(02)00243-X
Baker, M. (2016). Is there a reproducibility crisis? A Nature survey lifts the lid on how researchers view the ‘crisis’ rocking science and what they think will help. Nature, 533(7604), 452-455. doi: 10.1038/533452a
Bakker, M., van Dijk, A. & Wicherts, J. M. (2012). The Rules of the Game Called Psychological Science. Perspectives on Psychological Science, 7, 543-554. doi: 10.1177/1745691612459060
Blázquez, D., Botella, J., & Suero, M. (2017). The debate on the ego-depletion effect: Evidence from meta-analysis with the p-uniform method. Frontiers in Psychology, 8, 197. doi: 10.3389/fpsyg.2017.00197
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2010). A basic introduction to fixed-effects and random-effects models for meta-analysis. Research Synthesis Methods, 1, 97-111. doi: 10.1002/jrsm.12
Botella, J., & Eriksen, C. W. (1991). Pattern changes in rapid serial visual presentation tasks without strategic shifts. Bulletin of the Psychonomic Society, 29(2), 105-108.
Botella, J., & Gambara, H. (2006). El meta-análisis: una metodología de nues-tro tiempo. Infocop, 29 mayo.
Botella, J., & Sánchez-Meca, J. (2015). Meta-análisis en ciencias sociales y de la salud. Madrid: Síntesis.
Botella, J., Sepúlveda, A. R., Huang, H., & Gambara, H. (2013). A meta-analysis of the diagnostic accuracy of the SCOFF. Spanish Journal of Psy-chology, 16, e92, 1-8. doi:10.1017/sjp.2013.92
Botella, J., Suero, M., & Ximenez, C. (2012). Análisis de datos en psicología I. Ma-drid: Pirámide.
Botella, J., Ximénez, M. C., Revuelta, J., & Suero, M. (2006). Optimization of sample size in controlled experiments: the CLAST rule. Behavior Research Methods, Instruments & Computers, 38(1), 65-76. doi: 10.3758/BF03192751
Botella, J., & Zamora, Á. (2017). El meta-análisis: una metodología para la in-vestigación en educación. Educación XXI, 20(2), 17-38. doi: 10.5944/educXX1.18241
Cooper, H. M., Hedges, L.V., & Valentine, J.C. (Eds.) (2009). The handbook of research synthesis and meta-analysis (2ª ed.). Nueva York: Russell Sage Founda-tion.
Cumming, G. (2012). Understanding the new statistics: Effect sizes, confidence inter-vals, and meta-analysis. New York, NY: Routledge.
Cumming, G. (2014). The new statistics: why and how. Psychological Science, 25, 7–29. doi: 10.1177/0956797613504966
DeCoster, J., Sparks, E. A., Sparks, J. C., Sparks, G. G., & Sparks, C. W. (2015). Opportunistic biases: Their origins, effects, and an integrated so-lution. American Psychologist, 70(6), 499. doi: 10.1037/a0039191
Earp, B. D., & Trafimow, D. (2015). Replication, falsification, and the crisis of confidence in social psychology. Frontiers in psychology, 6, 621. doi: 10.3389/fpsyg.2015.00621
Frick, R. W. (1996). The appropriate use of null hypothesis testing. Psychological Methods, 1, 379-390. doi: 10.1037/1082-989X.1.4.379
Gigerenzer, G. (1993). The superego, the ego, and the statistical reasoning. En G. Keren y C. Lewis (eds). A Handbook for data analysis in the behavioral sci-ences: methodological issues. Hillsdale, NJ: Erlbaum.
Giner-Sorolla, R. (2012). Science or Art? How Aesthetic Standards Grease the Way Through the Publication Bottleneck but Undermine Science. Perspec-tives on Psychological Science, 7, 562-571. doi: 10.1177/1745691612457576
Glass, G. V. (1976). Primary, secondary and meta-analysis of research. Educa-tional Researcher, 5, 3–8. doi: 10.3102/0013189X005010003
Gøtzsche, P. C., Hróbjartsson, A., Marić, K., & Tendal, B. (2007). Data ex-traction errors in meta-analyses that use standardized mean differences. JAMA, 298(4), 430-437. doi:10.1001/jama.298.4.430
Hagger, M. S., & Chatzisarantis, N. L. D. (2016). A multilab preregistered rep-lication of the ego-depletion effect. Perspectives on Psychological Science, 11(4), 546-573. doi: 10.1177/1745691616652873
Harlow, L. L., Mulaik, S. A., & Steiger, J. H. (1997). What if there were no signifi-cance tests? (pp. 37-64). Mahwah, NJ: Lawrence Erlbaum Associates.
Hattie, J. A. C. (2009). Visible learning: A synthesis of 800+ meta-analyses on achievement. London: Routledge.
Hollenbeck, J. R., & Wright, P.M. (2017). Harking, Sharking, and Tharking: Making the case for post hoc analysis of scientific data. Journal of Manage-ment, 43(1), 5-18. doi: 10.1177/0149206316679487
Ioannidis, J. P. (2005). Why most published research findings are false. PLoS medicine, 2(8), e124. doi: 10.1371/journal.pmed.0020124
Ioannidis, J. (2016). The mass production of redundant, misleading, and con-flicted systematic reviews and meta‐analyses. The Milbank Quarterly, 94(3), 485-514. doi: 10.1111/1468-0009.12210
Ioannidis, J. P., & Trikalinos, T. A. (2007). An exploratory test for an excess of significant findings. Clinical trials, 4(3), 245-253. doi: 10.1177/1740774507079441
John, L., Loewenstein, G. F., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth-telling. Psychologi-cal Science, 23, 524–32. doi: 10.1177/0956797611430953
Klein, O., Doyen, S., Leys, C., Magalhães de Saldanha da Gama, P. A., Miller, S., Questienne, L. & Axel Cleeremans, A. (2012). Low Hopes, High Ex-pectations: Expectancy Effects and the Replicability of Behavioral Exper-iments. Perspectives on Psychological Science, 7, 572-584. doi: 10.1177/1745691612463704
Koole, S. L., & Lakens, D. (2012). Rewarding replications: A sure and simple way to improve psychological science. Perspectives on Psychological Science, 7(6), 608-614. doi: 10.1177/1745691612462586
Lakens, D., Hilgard, J., & Staaks, J. (2016). On the reproducibility of meta-analyses: Six practical recommendations. BMC psychology, 4(1), 24. doi: 10.1186/s40359-016-0126-3
Lash, T. L. (2017). The harm done to reproducibility by the culture of null hy-pothesis significance testing. American Journal of Epidemiology, 186(6), 627-635. doi: 10.1093/aje/kwx261
Lipsey, M. W., & Wilson, D. B. (1993). The efficacy of psychological, educa-tional, and behavioral treatment: Confirmation from meta-analysis. Amer-ican psychologist, 48(12), 1181-1209. doi: 10.1037/0003-066X.48.12.1181
Makel, M. C., Plucker, J. A., & Hegarty, B. (2012). Replications in Psychology Research: How Often Do They Really Occur? Perspectives on Psychological Science, 7, 537-542. doi: 10.1177/1745691612460688
Nickerson, R. S. (2000). Null hypothesis significance testing: a review of an old and continuing controversy. Psychological Methods, 5, 241-301. doi: 10.1037/1082-989X.5.2.241
Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia: II. Restructur-ing incentives and practices to promote truth over publishability. Perspec-tives on Psychological Science, 7, 615–631. doi: 10.1177/1745691612459058
Open Science Collaboration. (2015). Estimating the reproducibility of psycho-logical science. Science, 349(6251), aac4716. doi: 10.1126/science.aac4716
Pashler, H., & Harris, C. R. (2012). Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science, 7(6), 531-536. doi: 10.1177/1745691612463401
Pashler, H., & Wagenmakers, E.J. (2012). Editors’ Introduction to the Special Section on Replicability in Psychological Science: A Crisis of Confidence? Perspectives on Psychological Science, 7, 528-530. doi: 10.1177/1745691612465253
Quiles-Marcos, Y., Quiles-Sebastián, M., Pamies-Aubalat, L., Botella, J., & Treasure, J. (2013). Peer and family influence in eating disorders: A meta-analysis. European Psychiatry, 28(4), 199-206. doi:10.1016/j.eurpsy.2012.03.005
Rosenthal, R. 1979. The “file drawer problem” and tolerance for null results. Psychological Bulletin, 86, 638–641. doi: 10.1037/0033-2909.86.3.638
Rothstein, H. R., Sutton, A. J., & Borenstein, M. (Eds.) (2005). Publication bias in meta-analysis: Prevention, assessment, and adjustments. Nueva York: Wiley.
Schmidt, F. L., & Oh, I. S. (2016). The crisis of confidence in research find-ings in psychology: Is lack of replication the real problem? Or is it some-thing else? Archives of Scientific Psychology, 4(1), 32-37. doi: 10.1037/arc0000029
Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995). Publication deci-sions revisited: The effect of the outcome of statistical tests on the deci-sion to publish and vice versa. American Statistician, 49, 108–112. doi: 10.2307/2684823
Tryon, W. W. (2001). Evaluating statistical difference, equivalence, and inde-terminacy using inferential confidence intervals; an integrated alternative method of conducting null hypothesis statistical tests. Psychological Meth-ods, 6, 371-386. doi: 10.1037/1082-989X.6.4.371
Yong, E. (2012). In the wake of high-profile controversies, psychologists are facing up to problems with replication. Nature, 485, 298-300. doi: 10.1038/485298a
The works published in this journal are subject to the following terms:
1. The Publications Service of the University of Murcia (the publisher) retains the property rights (copyright) of published works, and encourages and enables the reuse of the same under the license specified in paragraph 2.
© Servicio de Publicaciones, Universidad de Murcia, 2022
2. The works are published in the online edition of the journal under a Creative Commons Reconocimiento-CompartirIgual 4.0 (legal text). You can copy, use, distribute, transmit and publicly display, provided that: i) you cite the author and the original source of publication (journal, editorial and URL of the work), ii) are not used for commercial purposes, iii ) mentions the existence and specifications of this license.
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
3. Conditions of self-archiving. Is allowed and encouraged the authors to disseminate electronically pre-print versions (version before being evaluated and sent to the journal) and / or post-print (version reviewed and accepted for publication) of their works before publication, as it encourages its earliest circulation and diffusion and thus a possible increase in its citation and scope between the academic community. RoMEO Color: Green.
