A journey around alpha and omega to estimate internal consistency reliability

Carme Viladrich, Ariadna Angulo-Brunet, Eduardo Doval

Abstract


Based on recent psychometric developments, this paper presents a conceptual and practical guide for estimating internal consistency reliability of measures obtained as item sum or mean. The internal consistency reliability coefficient is presented as a by-product of the measurement model underlying the item responses. A three-step procedure is proposed for its estimation, including descriptive data analysis, test of relevant measurement models, and computation of internal consistency coefficient and its confidence interval. Provided formulas include: (a) Cronbach’s alpha and omega coefficients for unidimensional measures with quantitative item response scales, (b) coefficients ordinal omega, ordinal alpha and nonlinear reliability for unidimensional measures with dichotomic and ordinal items, (c) coefficients omega and omega hierarchical for essentially unidimensional scales presenting method effects. The procedure is generalized to weighted sum measures, multidimensional scales, complex designs with multilevel and/or missing data and to scale development. Four illustrative numerical examples are fully explained and the data and the R syntax are provided.


Keywords


reliability; internal consistency; coefficient alpha; coefficient omega; congeneric measures; tau-equivalent measures; confirmatory factor analysis

Full Text:

PDF (Español) PDF

References


Abad, F. J., Olea, J., Ponsoda, V., & García, C. (2011). Medición en ciencias sociales y de la salud [Measurement in social and health sciences]. Madrid: Síntesis.

American Psychological Association (2010). Publication manual of the American Psychological Association. (6th ed.). Washington, DC

Behrens, J. T., DiCerbo, K. E., Yel, N., & Levy, R. (2012). Exploratory Data Analysis. In Handbook of Psychology, Second Edition. John Wiley & Sons, Inc. doi:10.1002/9781118133880.hop202002

Bentler, P. M. (2009). Alpha, dimension-free, and model-based internal consistency reliability. Psychometrika, 74(1), 137–143. doi:10.1007/s11336-008-9100-1

Bentler, P. M. (2016). Specificity-enhanced reliability coefficients. Psychological Methods, 0. Advance online publication. doi:10.1037/met0000092

Birnbaum, A. (1968). Some latent trait models and their use in inferring a examinee’s ability. In F. M. Lord & M. R. Novick (Eds.), Statistical Theories of Mental Test Scores (pp. 397–479). Reading, MA: Addison-Wesley.

Black, R. A., Yang, Y., Beitra, D., & McCaffrey, S. (2015). Comparing fit and reliability estimates of a psychological instrument using second-order CFA, bifactor, and essentially tau-equivalent (coefficient alpha) Models via AMOS 22. Journal of Psychoeducational Assessment, 33(5), 451–472. doi:10.1177/0734282914553551

Bovaird, J. A., & Koziol, N. A. (2012). Measurement models for ordered-categorical indicators. In Handbook of Structural Equation Modeling (pp. 495–511). New York, NY: The Guilford Press.

Bollen, K. (1980). Issues in the comparative measurement of political democracy. American Sociological Review, 45, 370–390.

Brown, T. A. (2015). Confirmatory factor analysis for applied research. 2nd Ed. London: The Guilford Press.

Brunner, M., Nagy, G., & Wilhelm, O. (2012). A tutorial on hierarchically structured constructs. Journal of Personality, 80(4), 796–846. doi:10.1111/j.1467-6494.2011.00749.x

Cheng, Y., Yuan, K. H., & Liu, C. (2012). Comparison of reliability measures under factor analysis and item response theory. Educational and Psychological Measurement, 72(1), 52–67. doi:10.1177/0013164411407315

Cho, E. (2016). Making Reliability Reliable: A Systematic approach to reliability coefficients. Organizational Research Methods, 19(4), 651–682. doi:10.1177/1094428116656239

Conway, J. M., & Lance, C. E. (2010). What reviewers should expect from authors regarding common method bias in organizational research. Journal of Business and Psychology, 25(3), 325–334. doi:10.1007/s10869-010-9181-6

Cronbach, L. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297-334.

Cronbach, L. J., & Shavelson, R. J. (2004). My current thoughts on coefficient alpha and successor procedures. Educational and Psychological Measurement, 64(3), 391–418. doi:10.1177/0013164404266386

Crutzen, R., & Peters, G. (2015). Scale quality: alpha is an inadequate estimate and factor-analytic evidence is needed first of all. Health Psychology Review, 1–6. doi:10.1080/17437199.2015.1124240

Davenport, E. C., Davison, M. L., Liou, P. Y., & Love, Q. U. (2016). Easier said than done: rejoinder on Sijtsma and on Green and Yang. Educational Measurement: Issues and Practice, 35(1), 6–10. doi:10.1111/emip.12106

Deng, L., & Chan, W. (2016). Testing the difference between reliability coefficients alpha and omega. Educational and Psychological Measurement, online, 1–19. doi:10.1177/0013164416658325

Dimitrov, D. M. (2003). Reliability and true-score measures of binary items as a function of their Rasch difficulty parameter. Journal of Applied Measurement, 4(3), 222–233. doi:10.1177/0146621603258786

Dunn, T. J., Baguley, T., & Brunsden, V. (2014). From alpha to omega: a practical solution to the pervasive problem of internal consistency estimation. British Journal of Psychology, 105(3), 399–412. doi:10.1111/bjop.12046

Elosua, P., & Zumbo, B. D. (2008). Reliability coefficients for ordinal response scales. Psicothema, 20(4), 896–901.

Enders, C. K. (2010). Applied missing data analysis. New York, NY: The Guilford Press.

Enders, C. K. (2013). Dealing with missing data in developmental research. Child Development Perspectives, 7(1), 27–31. doi:10.1111/cdep.12008

Ferrando, P. J., & Lorenzo-Seva, U. (2014). El análisis factorial exploratorio de los ítems: Algunas consideraciones adicionales [Exploratory item factor analysis: some additional considerations] Anales de Psicologia, 30(3), 1170–1175. doi:10.6018/analesps.30.3.199991

Ferrando, P. J., & Lorenzo-Seva, U. (2017). Program FACTOR at 10: origins, development and future directions. Psicothema, 29(2), 236–240. https://doi.org/10.7334/psicothema2016.304

Forero, C. G., Maydeu-Olivares, A., & Gallardo-Pujol, D. (2009). Factor analysis with ordinal indicators: A Monte Carlo study comparing DWLS and ULS estimation. Structural Equation Modeling, 16(4), 625–641. doi:10.1080/10705510903203573

Gabler, S., & Raykov, T. (2017). Evaluation of maximal reliability for unidimensional measuring instruments with correlated errors. Structural Equation Modeling: A Multidisciplinary Journal, 24(1), 104-111. Advance online publication. doi:10.1080/10705511.2016.1159916

Gadermann, A. M., Guhn, M., & Zumbo, B. D. (2012). Estimating ordinal reliability for likert-type and ordinal item response data: A conceptual, empirical, and practical guide. Practical Assessment, Research and Evaluation, 17(3), 1–13.

Gignac, G. E. (2014). On the inappropriateness of using items to calculate total scale score reliability via coefficient alpha for multidimensional scales. European Journal of Psychological Assessment, 30(2), 130–139. doi:10.1027/1015-5759/a000181

Graham, J. (2006). Congeneric and (essentially) tau-equivalent estimates of score reliability. What they are and how to use them. Educational and Psychological Measurement, 66(6), 930–944. doi: 10.1177/0013164406288165.

Graham, J. W. (2009). Missing data analysis: Making it work in the real world. Annual Review of Psychology, 60, 549–76. doi:10.1146/annurev.psych.58.110405.085530

Green, S. B., & Yang, Y. (2009). Reliability of summed item scores using structural equation modeling: An alternative to coefficient alpha. Psychometrika, 74(1), 155–167. doi:10.1007/s11336-008-9099-3

Green, S. B., & Yang, Y. (2015). Evaluation of dimensionality in the assessment of internal consistency reliability: coefficient alpha and omega coefficients. Educational Measurement: Issues and Practice, 34(4), 14–20. doi:10.1111/emip.12100

Green, S. B., Yang, Y., Alt, M., Brinkley, S., Gray, S., Hogan, T., & Cowan, N. (2016). Use of internal consistency coefficients for estimating reliability of experimental task scores. Psychonomic Bulletin & Review, 23(3), 750–763. doi:10.3758/s13423-015-0968-3

Gu, F., Little, T. D., & Kingston, N. M. (2013). Misestimation of reliability using coefficient alpha and structural equation modeling when assumptions of tau-equivalence and uncorrelated errors are violated. Methodology, 9(1), 30–40. doi:10.1027/1614-2241/a000052

Hancock, G. R., & Mueller, R. O. (2001). Rethinking construct reliability within latent variable systems. In R. Cudeck, S. du Toit, & D. Sorbom (Eds.), Structural Equation Modeling: Present and future (pp. 195–216). Lincolnwood, IL: Scientific Software International, Inc.

Hancock, G. R., & Mueller, R. O. (Eds.). (2013). Structural equation modeling. A second course (2nd ed.). Charlotte, NC: Information Age Publishing.

Hoyle, R. H. (Ed.). (2012). Handbook of Structural equation modeling. New York, NY: The Guilford Press.

Huggins-Manley, A. C., & Han, H. (2017). Assessing the sensitivity of weighted least squares model fit indexes to local dependence in item response theory models. Structural Equation Modeling: A Multidisciplinary Journal, 24(3), 331–340. doi:10.1080/10705511.2016.1247355

Izquierdo, I., Olea, J., & Abad, F. J. (2014). Exploratory factor analysis in validation studies: Uses and recommendations. Psicothema, 26(3), 395–400. doi:10.7334/psicothema2013.349

Jöreskog, K. G. (1971). Statistical analysis of sets of congeneric tests. Psychometrika, 36(2), 109–133. doi:10.1007/s00170-004-2446-3

Kelley, K., & Pornprasertmanit, S. (2016). Confidence intervals for population reliability coefficients : Evaluation of methods, recommendations, and software for composite measures. Psychological Methods, 21(1), 69–92. doi:10.1037/a0040086

Kim, S., & Feldt, L. S. (2010). The estimation of the IRT reliability coefficient and its lower and upper bounds, with comparisons to CTT reliability statistics. Asia Pacific Education Review, 11(2), 179–188. doi:10.1007/s12564-009-9062-8

Lance, C. E., Dawson, B., Birkelbach, D., & Hoffman, B. J. (2010). Method effects, measurement error, and substantive conclusions. Organizational Research Methods, 13(3), 435–455. doi:10.1177/1094428109352528

Little, T. D., Rhemtulla, M., Gibson, K., & Schoemann, A. M. (2013). Why the items versus parcels controversy needn’t be one. Psychological Methods, 18(3), 285–300. doi:10.1037/a0033266

Lloret-Segura, S., Ferreres-Traver, A., Hernández-Baeza, A., & Tomás-Marco, I. (2014). El análisis factorial exploratorio de los ítems: una guía práctica, revisada y actualizada. [Exploratory item factor analysis: A practical guide revised and up-dated] Anales de Psicología, 30(3), 1151–1169. doi:10.6018/analesps.30.3.199361

Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison Wesley.

Malone, P. S., & Lubansky, J. B. (2012). Preparing data for structural equation modeling: doing your homework. In R. H. Hoyle (Ed.), Handbook of Structural Equation Modeling (pp. 263–276). New York, NY: The Guilford Press.

Mansolf, M., & Reise, S. P. (2016). Exploratory bifactor analysis: The Schmid-Leiman orthogonalization and Jennrich-Bentler analytic rotations. Multivariate Behavioral Research, 51(5), 698–717. doi:10.1080/00273171.2016.1215898

Marsh, H. W. (1996). Positive and negative global self-esteem: a substantively meaningful distinction or artifactors? Journal of Personality and Social Psychology, 70(4), 810–819. doi:10.1037/0022-3514.70.4.810

Marsh, H. W., Lüdtke, O., Muthén, B., Asparouhov, T., Morin, A. J. S., Trautwein, U., & Nagengast, B. (2010). A new look at the big five factor structure through exploratory structural equation modeling. Psychological Assessment, 22(3), 471–91. doi:10.1037/a0019227

Marsh, H. W., Lüdtke, O., Nagengast, B., Morin, A. J. S., & Von Davier, M. (2013). Why item parcels are (almost) never appropriate: Two wrongs do not make a right--camouflaging misspecification with item parcels in CFA models. Psychological Methods, 18, 257–84. doi:10.1037/a0032773

Maydeu-Olivares, A., & Coffman, D. L. (2006). Random intercept item factor analysis. Psychological Methods, 11(4), 344–362. doi:10.1037/1082-989X.11.4.344

Maydeu-Olivares, A., Fairchild, A. J., & Hall, A. G. (2017). Goodness of fit in item factor analysis: effect of the number of response alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 0(0), 1–11. doi:10.1080/10705511.2017.1289816

McCrae, R. R. (2014). A more nuanced view of reliability: specificity in the trait hierarchy. Personality and Social Psychology Review, 19(2), 97–112. doi:10.1177/1088868314541857

McDonald, R. P. (1999). Test theory: A unified treatment. Mahwah, NJ: Lawrence Erlbaum Associates.

McNeish, D. (2017). Thanks coefficient alpha, we’ll take it from here. Psychological Methods, 0. Advance online publication. doi: 10.1037/met0000144

Muñiz, J. (1992). Teoría clásica de los tests [Classical test theory]. Madrid: Pirámide.

Muthén, B. (1984). A general structural equation model with dichotomous, ordered categorical, and continuous latent variable indicators. Psychometrika, 49(1), 115–132.

Muthén, L. K., & Muthén, B. O. (2017). Mplus User’s Guide (Eighth Edition). Los Angeles, CA: Muthén & Muthén.

Muthén & Muthén (n.d.). Chi-Square difference testing using the Satorra-Bentler scaled Chi-Square. Retrieved June 19, 2017 from https://www.statmodel.com/chidiff.shtml

Napolitano, C. M., Callina, K. S., & Mueller, M. K. (2013). Comparing alternate approaches to calculating reliability for dichotomous data: The sample case of adolescent selection, optimization, and compensation. Applied Developmental Science, 17(3), 148–151. doi:10.1080/10888691.2013.804372

Ntoumanis, N., Mouratidis, T., Ng, J. Y. Y., & Viladrich, C. (2015). Advances in quantitative analyses and their implications for sport and exercise psychology research. In S. Hanton & S. D. Mellalieu (Eds.), Contemporary advances in sport psychology: A review. (pp. 226–257). London: Routledge.

Nunnally, J. C. (1978). Psychometric theory. New York, NY: McGrawHill.

Padilla, M. A., & Divers, J. (2016). A comparison of composite reliability estimators: coefficient omega confidence intervals in the current literature. Educational and Psychological Measurement, 76(3), 436–453. doi:10.1177/0013164415593776

R Core Team (2016). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. Retrived from https://www.R-project.org/.

Raykov, T. (1997). Estimation of composite reliability for congeneric measures. Applied Psychological Measurement, 21(2), 173–184. doi: 0803973233

Raykov, T. (1998). Coefficient alpha and composite reliability with interrelated nonhomogeneous items. Applied Psychological Measurement, 22(4), 375–385. doi:10.1177/014662169802200407

Raykov, T. (2001). Bias of coefficient alpha for fixed congeneric measures with correlated errors. Applied Psychological Measurement, 25(1), 69–76. doi:10.1177/01466216010251005

Raykov, T. (2004). Point and interval estimation of reliability for multiple-component measuring instruments via linear constraint covariance structure modeling. Structural Equation Modeling, 11(3), 452–483. doi:10.1207/s15328007sem1103

Raykov, T. (2007). Reliability if deleted, not “alpha if deleted”: Evaluation of scale reliability following component deletion. The British Journal of Mathematical and Statistical Psychology, 60(2), 201–216. doi: 10.1348/000711006X115954

Raykov, T. (2012). Scale construction and development using structural equation modeling. In R. H. Hoyle (Ed.), Handbook of Structural Equation Modeling (pp. 472–492). New York, NY: Guilford Press.

Raykov, T., Dimitrov, D. M., & Asparouhov, T. (2010). Evaluation of scale reliability with binary measures using latent variable modeling. Structural Equation Modeling: A Multidisciplinary Journal, 17(2), 265–279. doi:10.1080/10705511003659417

Raykov, T., Gabler, S., & Dimitrov, D. M. (2016). Maximal reliability and composite reliability: examining their difference for multicomponent measuring instruments using latent variable modeling. Structural Equation Modeling: A Multidisciplinary Journal, 23(3), 384–391. doi:10.1080/10705511.2014.966369

Raykov, T., & Marcoulides, G. A. (2012). Evaluation of validity and reliability for hierarchical scales using latent variable modeling. Structural Equation Modeling: A Multidisciplinary Journal, 19(3), 495–508. doi:10.1080/10705511.2012.687675

Raykov, T., & Marcoulides, G. A. (2014). Scale reliability evaluation with heterogeneous populations. Educational and Psychological Measurement, 75(5), 875–892. doi:10.1177/0013164414558587

Raykov, T., & Marcoulides, G. A. (2015). A direct latent variable modeling based method for point and interval estimation of coefficient alpha. Educational and Psychological Measurement, 75(1), 146–156. doi:10.1177/0013164414526039

Raykov, T., & Marcoulides, G. A. (2016a). Scale reliability evaluation under multiple assumption violations. Structural Equation Modeling: A Multidisciplinary Journal, 23(2), 302–313. doi:10.1080/10705511.2014.938597

Raykov, T., & Marcoulides, G. A. (2016b). On Examining specificity in latent construct indicators. Structural Equation Modeling: A Multidisciplinary Journal, 23(6), 845-855. doi:10.1080/10705511.2016.1175947

Raykov, T., & Pohl, S. (2013). Essential unidimensionality examination for multicomponent scales: an interrelationship decomposition approach. Educational and Psychological Measurement, 73(4), 581–600. doi:10.1177/0013164412470451

Raykov, T., West, B. T., & Traynor, A. (2015). Evaluation of coefficient alpha for multiple-component measuring instruments in complex sample designs. Structural Equation Modeling: A Multidisciplinary Journal, 22(3), 429–438. doi:10.1080/10705511.2014.936081

Reise, S. P. (2012). The rediscovery of bifactor measurement models. Multivariate Behavioral Research, 47(5), 667–696. doi:10.1080/00273171.2012.715555

Revelle, W. (2016). psych: Procedures for personality and psychological research. R package version 1.6.4. North-western University, Evanston. Retrieved June 19, 2017 from http://cran.r-project.org/web/packages/psych/.

Revelle, W., & Zinbarg, R. E. (2009). Coefficient alpha, beta, omega, and the GLB: Comment on Sitjsma. Psychometrika, 74(1), 145–154.

Rhemtulla, M., Brosseau-Liard, P. É., & Savalei, V. (2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods, 17(3), 354–373. doi:10.1037/a0029315

Rodriguez, A., Reise, S. P., & Haviland, M. G. (2016). Evaluating bifactor models: Calculating and interpreting statistical indices. Psychological Methods, 21(2), 137–150. doi:10.1037/met0000045

Rosseel, Y. (2012). lavaan: An R Package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36.

Sass, D. a., Schmitt, T. a., & Marsh, H. W. (2014). Evaluating model fit with ordered categorical data within a measurement invariance framework: A comparison of estimators. Structural Equation Modeling: A Multidisciplinary Journal, 21(2), 167–180. doi:10.1080/10705511.2014.882658

semTools Contributors. (2016). semTools: Useful tools for structural equation modeling. R package version 0-4-11.

Sijtsma, K. (2009). On the use, the misuse, and the very limited usefulness of Cronbach’s alpha. Psychometrika, 74(1), 107–120. doi:10.1007/s11336-008-9101-0

Sijtsma, K. (2015). Delimiting coefficient alpha from internal consistency and unidimensionality. Educational Measurement: Issues and Practice, 34(4), 10–13.

Spector, P. E. (2006). Method variance in organiztional research. Truth or urban legend ? Organizational Research Methods, 9(2), 221–232. doi:1094428105284955

Stout, W. (1987). A nonparametric approach for assessing latent trait unidimensionality. Psychometrika, 52(4), 589–617. doi:10.1007/BF02294821

Wickham, H. (2007). Reshaping data with the reshape package. Journal of Statistical Software, 21(12), 1-20. Retrieved from http://www.jstatsoft.org/v21/i12/.

Yang, Y., & Green, S. B. (2010). A note on structural equation modeling estimates of reliability. Structural Equation Modeling: A Multidisciplinary Journal, 17(1), 66–81. doi:10.1080/10705510903438963

Yang, Y., & Green, S. B. (2011). Coefficient alpha: A reliability coefficient for the 21st Century? Journal of Psychoeducational Assessment, 29(4), 377–392. doi:10.1177/0734282911406668

Yang, Y., & Green, S. B. (2015). Evaluation of structural equation modeling estimates of reliability for scales with ordered categorical items. Methodology, 11(1), 23–34. doi:10.1027/1614-2241/a000087

Zhang, Z., & Yuan, K.-H. (2016). Robust coefficients alpha and omega and confidence intervals with outlying observations and missing data: Methods and software. Educational and Psychological Measurement, 76(3), 387–411. doi:10.1177/0013164415594658

Zinbarg, R. E., Revelle, W., Yovel, I., & Li, W. (2005). Cronbach’s α, Revelle’s β, and Mcdonald’s ωH: their relations with each other and two alternative conceptualizations of reliability. Psychometrika, 70(1), 123–133. doi:10.1007/s11336-003-0974-7

Zumbo, B. D., Gadermann, A. M., & Zeisser, C. (2007). Ordinal versions of coefficients alpha and theta for Likert rating scales. Journal of Modern Applied Statistical Methods, 6(1), 21–29. doi:10.1107/S0907444909031205




DOI: http://dx.doi.org/10.6018/analesps.33.3.268401

Refbacks

  • There are currently no refbacks.


Copyright (c) 2017 Servicio de Publicaciones, Universidad de Murcia (Spain)

Open AccessSello de Calidad FECyT 2013Thomson-Reuters-JCRScielo-Españadoajscimago