Issues and possible solutions in cognitive diagnosis modeling applications: The case of a large-scale educational assessment in Mexico
Supporting Agencies
- MICIU/AEI/10.13039/501100011033 and ERDF/EU under the project “Computerized adaptive tests based on new assessment formats” (reference: PID2022-137258NB-I00)
- UAM IIC Chair on Psychometric Models and Applications
Abstract
Cognitive diagnosis modeling (CDM) is a proposed framework for the creation and analysis of measurement tools that originated in the field of education and have extended to other areas of interest in psychology. These models have received a lot of attention in recent years, resulting in an abundance of theoretical contributions. However, there is still a shortage of studies applying these methodologies to real data. It is essential to evaluate the procedures in their natural context of application and to provide solutions tailored to the problems that may arise in practice. The purpose of this study is to apply CDM to a large-scale evaluation that assesses high school teachers in Mexico, which was designed based on a CDM framework. Five issues are identified that arose in the analysis of these data and provide guidelines on how to explore and find solutions to these issues with implementation in R. Besides discussing psychometric solutions, this paper also emphasizes the importance of consulting with content experts. By highlighting potential challenges that can arise even when all necessary elements for a large-scale CDM educational assessment are in place, this study aims to guide future empirical applications and methodological proposals.
Downloads
-
Abstract0
-
pdf0
-
xml0
References
Alonso-Tapia, J., Ruiz, M. Á., & Huertas, J. A. (2020). Diferencias en el clima motivacional en el aula: causas, efectos e implicaciones para la formación docente. Un estudio multinivel. Annals of Psychology, 36(1), 122–133. https://doi.org/10.6018/analesps.337911
Chen, Y., Culpepper, S. A., & Chen, Y. (2023). Bayesian inference for an unknown number of attributes in restricted latent class models. Psychometrika, 88(2), 613-635. https://doi.org/10.1007/s11336-022-09900-7
de la Torre, J. (2011). The generalized DINA model framework. Psychometrika, 76(2), 179–199. https://doi.org/10.1007/s11336-011-9207-7
de la Torre, J., & Douglas, J. A. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69(3), 333-353. https://psycnet.apa.org/doi/10.1007/BF02295640
de la Torre, J., & Sorrel, M. A. (2023). Cognitive diagnosis models. In F. Ashby, H. Colonius, & E. Dzhafarov (Eds.), New Handbook of Mathematical Psychology (Cambridge Handbooks in Psychology, pp. 385-420). Cambridge: Cambridge University Press. https://doi.org/10.1017/9781108902724.010
Gao, M., Miller, M. D., & Liu, R. (2017). The impact of Q-matrix misspecification and model misuse on classification accuracy in the generalized DINA model. Journal of Measurement and Evaluation in Education and Psychology, 8, 391–403. https://doi.org/10.21031/epod.332712
García, P. E., Olea, J., & de la Torre, J. (2014). Application of cognitive diagnosis models to competency-based situational judgment tests. Psicothema, 26(3), 372-377.
García-Bacete, F, J., & Rosel-Remírez, J. F. (2021). Spanish validation of the Questionnaire on Teacher Interaction in the upper grades of primary education (QTI-P) and how this interaction influences academic performance. Annals of Psychology, 37(1), 101-113. https://dx.doi.org/10.6018/analesps.37.1.415111
Gierl, M. J., & Cui, Y. (2008). Defining characteristics of diagnostic classification models and the problem of retrofitting in cognitive diagnostic assessment. Measurement: Interdisciplinary Research and Perspectives, 6(4), 263–268. https://doi.org/10.1080/15366360802497762
Gierl, M. J., Alves, C., & Majeau, R. T. (2010). Using the attribute hierarchy method to make diagnostic inferences about examinees’ knowledge and skills in mathematics: An operational implementation of cognitive diagnostic assessment. International Journal of Testing, 10, 318–341. https://doi.org/10.1080/15305058.2010.509554
Gu, Y., & Xu, G. (2021). Sufficient and necessary conditions for the identifiability of the Q-matrix. Statistica Sinica, 31, 449-472. https://www.jstor.org/stable/26969691
Junker, B.W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258–272. https://doi.org/10.1177/01466210122032064
Liu, R., Huggins-Manley, A. C., & Bulut, O. (2018). Retrofitting diagnostic classification models to responses from IRT-based assessment forms. Educational and psychological measurement, 78(3), 357-383. https://doi.org/10.1177/0013164416685599
Liu, Y., Tian, W., & Xin, T. (2016). An Application of M2 Statistic to Evaluate the Fit of Cognitive Diagnostic Models. Journal of Educational and Behavioral Statistics, 41, 3-26. https://doi.org/10.3102/1076998615621293
Ma, W., & de la Torre, J. (2020). GDINA: An R package for cognitive diagnosis modeling. Journal of Statistical Software, 93(14), 1-26. https://doi.org/10.18637/jss.v093.i14
Ma, W., Iaconangelo, C., & de la Torre, J. (2016). Model similarity, model selection, and attribute classification. Applied Psychological Measurement, 40(3), 200–217. https://doi.org/10.1177/0146621615621717
Ma, W., Minchen, N., & de la Torre, J. (2020). Choosing between CDM and unidimensional IRT: The proportional reasoning test case. Measurement: Interdisciplinary Research and Perspectives, 18(2), 87-96. https://doi.org/10.1080/15366367.2019.1697122
Maydeu-Olivares, A., & Joe, H. (2006). Limited information goodness-of-fit testing in multidimensional contingency tables. Psychometrika, 71, 713-732. https://doi.org/10.1007/s11336-005-1295-9
Moreno-Murcia, J. A., Huéscar Hernández, E., León, J., Fin, G., Nodari Júnior, R. J., Valero-Valenzuela, A., Tristán, J., Gastélum-Cuadras, G., Zueck Enríquez, M. C., Vargas Vitoria, R., Cid, L., Monteiro, D., & Teixeira, D. (2024). Motivación para aprender: un estudio internacional multinivel sobre la autonomía de los estudiantes y el énfasis de los docentes en la utilidad del contenido. Annals of Psychology, 40(2), 265–271. https://doi.org/10.6018/analesps.571161
Nájera, P., Sorrel, M. A., & Abad, F. J. (2024). cdmTools: Useful Tools for Cognitive Diagnosis Modeling. R package version 1.0.5. https://cran.r-project.org/web/packages/cdmTools/
Nájera, P., Sorrel, M. A., de la Torre, J., & Abad, F. J. (2021). Balancing fit and parsimony to improve Q‐matrix validation. British Journal of Mathematical and Statistical Psychology, 74, 110-130.
R Core Team. (2024). R: A Language and Environment for Statistical Computing. R foundation for statistical computing, Vienna, Austria. https://www.R-project.org/
Ravand, H., & Robitzsch, A. (2018). Cognitive diagnostic model of best choice: a study of reading comprehension. Educational Psychology, 38(10), 1255–1277. https://doi.org/10.1080/01443410.2018.1489524
Ren, H., Xu, N., Lin, Y., Zhang, S., & Yang, T. (2021). Remedial teaching and learning from a cognitive diagnostic model perspective: Taking the data distribution characteristics as an example. Frontiers in Psychology, 12. https://doi.org/10.3389/fpsyg.2021.628607
Rupp, A., & Templin, J. (2008). The effects of Q-matrix misspecification on parameter estimates and classification accuracy in the DINA model. Educational and Psychological Measurement, 68, 78–96. https://doi.org/10.1177/0013164407301545
Sanz, S., Kreitchmann, R. S., Nájera, P., Moreno, J. D., Martínez-Huertas, J. Á., & Sorrel, M. A. (2023). FoCo: A Shiny app for formative assessment using cognitive diagnosis modeling. Psicología Educativa. Revista de los Psicólogos de la Educación, 29(2), 149-158.
Secretaría de Educación Pública & Unidad del Sistema para la Carrera de las Maestras y los Maestros. (December 2019). Marco para la excelencia en la enseñanza y la gestión escolar en la Educación Media Superior: Perfiles profesionales, criterios e indicadores para docentes, técnicos docentes y personal con funciones de dirección y de supervisión. Ciclo Escolar 2020-2021.
Sessoms, J., & Henson, R. A. (2018). Applications of diagnostic classification models: A Literature review and critical commentary. Measurement: Interdisciplinary Research and Perspectives, 16(1), 1–17. https://doi.org/10.1080/15366367.2018.1435104
Shi, Q., Ma, W., Robitzsch, A., Sorrel, M. A., & Man, K. (2021). Cognitively diagnostic analysis using the G-DINA model in R. Psych, 3(4), 812–835. https://doi.org/10.3390/psych3040052
Sorrel, M. A., Abad, F. J., & Nájera, P. (2021). Improving accuracy and usage by correctly selecting: The effects of model selection in cognitive diagnosis computerized adaptive testing. Applied Psychological Measurement, 45(2), 112-129. https://doi.org/10.1177/0146621620977682
Sorrel, M. A., Abad, F. J., Olea, J., de la Torre, J., & Barrada, J. R. (2017a). Inferential item-fit evaluation in cognitive diagnosis modeling. Applied Psychological Measurement, 41, 614–631.
Sorrel, M. A., de la Torre, J., Abad, F. J., & Olea, J. (2017b). Two-step likelihood ratio test for item-level model comparison in cognitive diagnosis models. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 13(Suppl 1), 39–47. https://doi.org/10.1027/1614-2241/a000131
Sorrel, M. A., Olea, J., Abad, F. J., de la Torre, J., Aguado, D., & Lievens, F. (2016). Validity and Reliability of Situational Judgement Test Scores: A New Approach Based on Cognitive Diagnosis Models. Organizational Research Methods, 19(3), 506-532. https://doi.org/10.1177/1094428116630065
Tan, Z., de la Torre, J., Ma, W., Huh, D., Larimer, M. E., & Mun, E. Y. (2022). A tutorial on cognitive diagnosis modeling for characterizing mental health symptom profiles using existing item responses. Prevention Science, 24, 480–492. https://doi.org/10.1007/s11121-022-01346-8
Tatsuoka, K. K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20, 345–354. https://doi.org/10.1111/j.1745-3984.1983.tb00212.x
Templin, J. L., & Henson, R. A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11(3), 287–305. https://doi.org/10.1037/1082-989X.11.3.287
Copyright (c) 2026 The authors
This work is licensed under a Creative Commons Attribution 4.0 International License.
About Copyright and Licensing, more details here.
