Meta-analysis: Confidence intervals and Prediction intervals
Abstract
Several types of intervals are usually employed in meta-analysis, a fact that has generated some confusion when interpreting them. Confidence intervals reflect the uncertainty related to a single number, the parametric mean effect size. Prediction intervals reflect the probable parametric effect size in any study of the same class as those included in a meta-analysis. Its interpretation and applications are different. In this article we explain their different nature and how they can be used to answer specific questions. Numerical examples are included, as well as their computation with the metafor R package.
Downloads
References
Blázquez‑Rincón, D., Sánchez‑Meca, J., Botella, J., & Suero, M. (2023). Heterogeneity estimation in meta‑analysis of standardized mean differences when the distribution of random effects departs from normal: A Monte Carlo simulation study. BMC Medical Research Methodology, 23(1), 19. https://doi.org/1 .1186/s12874-022-01809-0
Borenstein, M. (2019a). Common mistakes in meta-analysis. Englewood, NJ: Biostat inc.
Borenstein, M. (2019b). Heterogeneity in meta-analysis. In H. Cooper, L. V. Hedges, & J. C. Valentine (eds.), The Handbook of Research Synthesis and Meta-analysis, 3rd ed. (pp. 453-468). New York: Russell Sage Foundation.
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2017). Basics of meta-analysis: I2 is not an absolute measure of heterogeneity. Research Synthesis Methods, 8(1), 5-18. https://doi.org/1 .1002/jrsm.1230
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2021). Introduction to meta-analysis (2ª ed.). Chichester, UK: John Wiley and sons. [Chapter 17]
Botella, J., & Sánchez-Meca, J. (2015). [Meta-analysis in Social and Health Sciences] Meta-análisis en Ciencias Sociales y de la Salud. Madrid: Editorial Síntesis.
Hartung, J., & Knapp, G. (2001). On tests of the overall treatment effect in meta-analysis with normally distributed responses. Statistics in Medicine, 20(12), 1771-1782. https://doi.org/1 .1002/sim.791
Hedges, L. V., & Olkin, I. (1985). Statistical Methods for Meta-analysis. San Diego, CA: Academic Press.
Higgins, J. P. T., Thomas, J., Chandler, J., Cumpston, M., Li, T., Page, M. J., & Welch, V. A. (Eds) (2019). Cochrane handbook for systematic reviews of interventions. (2nd edition). Wiley.
Higgins, J. P., & Thompson, S. G. (2002). Quantifying heterogeneity in a meta‐analysis. Statistics in medicine, 21(11), 1539-1558. https://doi.org/1 .1002/sim.1186
Higgins, J. P., Thompson, S. G., & Spiegelhalter, D. J. (2009). A re‐evaluation of random‐effects meta‐analysis. Journal of the Royal Statistical Society: Series A (Statistics in Society), 172(1), 137-159. https://doi.org/1 .1111/j.1467-985X.2008.00552.x
Huedo-Medina, T., Sánchez-Meca, J., Marín-Martínez, F., & Botella, J. (2006). Assessing heterogeneity in meta-analysis: Q statistics or I2 index? Psychological Methods, 11, 193–206. https://doi.org/1 .1037/1082-989X.11.2.193
Hunter, J. E., & Schmidt, F. L. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage.
IntHout, J., Ioannidis, J. P., & Borm, G. F. (2014). The Hartung-Knapp-Sidik-Jonkman method for random effects meta-analysis is straightforward and considerably outperforms the standard DerSimonian-Laird method. BMC Medical Research Methodology, 14(25). http://www.biomedcentral.com/1471-2288/14/25
IntHout, J., Ioannidis, J. P., Rovers, M. M., & Goeman, J. J. (2016). Plea for routinely presenting prediction intervals in meta-analysis. BMJ open, 6(7), e010247. http://doi:1 .1136/bmjopen-2015-010247
Jackson, D., Law, M., Rücker, G., & Schwarzer, G. (2017). The Hartung-Knapp modification for random-effects meta-analysis: A useful refinement but are there any residual concerns? Statistics in Medicine, 36, 3923–3934. https://doi.org/1 .1002/sim.7411
Langan, D., Higgins, J. P., & Simmonds, M. (2017). Comparative performance of heterogeneity variance estimators in meta‐analysis: a review of simulation studies. Research Synthesis Methods, 8(2), 181-198. https://doi.org/1 .1002/jrsm.1198
Langan, D., Higgins, J. P., Jackson, D., Bowden, J., Veroniki, A. A., Kontopantelis, E., ... & Simmonds, M. (2019). A comparison of heterogeneity variance estimators in simulated random‐effects meta‐analyses. Research Synthesis Methods, 10(1), 83-98. https://doi.org/1 .1002/jrsm.1316
Partlett, C., & Riley, R.D. (2017). Random effects meta-analysis: Coverage performance of 95% confidence and prediction intervals following REML estimation. Statistics in Medicine, 36, 301-317. https://doi.org/1 .1002/sim.7140
Riley, R. D., Higgins, J. P., & Deeks, J. J. (2011). Interpretation of random effects meta-analyses. BMJ, 342. https://doi.org/1 .1136/bmj.d549
Sánchez-Meca, J., & Marín-Martínez, F. (2008). Confidence intervals for the overall effect size in random-effects meta-analysis. Psychological Methods, 13, 31-48. https://doi.org/1 .1037/1082-989X.13.1.31
Schmid, C. H., Stijnen, T., & White, I. (Eds.). (2021). Handbook of Meta-analysis. CRC Press.
Schmid, C. H, Carlin, B. P., & Welton, N. J. (2021). Bayesian Methods for Meta-analysis. En Handbook of Meta-Analysis (pp. 41-64). Chapman and Hall/CRC.
Sidik, K., & Jonkman, J. N. (2002). A simple confidence interval for meta-analysis. Statistics in Medicine, 21(21), 3153–3159. https://doi.org/1 .1002/sim.1262
Stijnen, T., White, I. R., & Schmid, C. H. (2021). Analysis of univariate study-level summary data using normal models. In Handbook of Meta-Analysis (pp. 41-64). Chapman and Hall/CRC. [Section 4.4.4.2]
Suero, M., Botella, J., & Durán, J. I. (2023). Methods for estimating the sampling variance of the standardized mean difference. Psychological Methods, 28(4), 895-904. https://doi.org/1 .1037/met0000446
Suero, M., Botella, J., Durán, J. I., & Blázquez-Rincón, D. (2023, September 12). Reformulating the meta-analytical random effects model as a mixture model. https://doi.org/1 .17605/OSF.IO/V2FDE
Veroniki, A. A., Jackson, D., Viechtbauer, W., Bender, R., Bowden, J., Knapp, G., ... & Salanti, G. (2016). Methods to estimate the between‐study variance and its uncertainty in meta‐analysis. Research Synthesis Methods, 7(1), 55-79. https://doi.org/1 .1002/jrsm.1164
Viechtbauer W. (2005). Bias and efficiency of meta-analytic variance estimators in the random-effects Model. Journal of Educational and Behavioral Statistics, 30, 261–293. https://doi.org/1 .3102/10769986030003261
Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36(3), 1-48. https://doi.org/1 .18637/jss.v036.i03
Viechtbauer, W. (2023). Package ‘metafor’. Unpublished document. University de Maastricht.
Viechtbauer, W., López-López, J. A., Sánchez-Meca, J., & Marín-Martínez, F. (2015). A comparison of procedures to test for moderators in mixed-effects meta-regression models. Psychological Methods, 20, 360-374. https://doi.org/1 .1037/met0000023
Copyright (c) 2024 Servicio de Publicaciones, University of Murcia (Spain)
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
The works published in this journal are subject to the following terms:
1. The Publications Service of the University of Murcia (the publisher) retains the property rights (copyright) of published works, and encourages and enables the reuse of the same under the license specified in paragraph 2.
© Servicio de Publicaciones, Universidad de Murcia, 2022
2. The works are published in the online edition of the journal under a Creative Commons Reconocimiento-CompartirIgual 4.0 (legal text). You can copy, use, distribute, transmit and publicly display, provided that: i) you cite the author and the original source of publication (journal, editorial and URL of the work), ii) are not used for commercial purposes, iii ) mentions the existence and specifications of this license.
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
3. Conditions of self-archiving. Is allowed and encouraged the authors to disseminate electronically pre-print versions (version before being evaluated and sent to the journal) and / or post-print (version reviewed and accepted for publication) of their works before publication, as it encourages its earliest circulation and diffusion and thus a possible increase in its citation and scope between the academic community. RoMEO Color: Green.
