Training program evaluation in non-standardized context: Comple-mentarity across Factorial and Multilevel Analysis to obtain construct validity evidences.
Supporting Agencies
- Ministerio de Innovación y Ciencia
- Fondo Nacional de Desarrollo Científico y Tecnológico de Chile
Abstract
Program evaluation is usually applied to non-standardized intervention contexts. This implies, among others, deficiencies of: a) validated theoretical models; b) non-standard measurement instruments; c) reliable measures. In this work, we show that Factor Analysis with polychoric correlations and Multilevel Analysis could be an adequate procedure to gain construct validity evidence in non-standard evaluative contexts, where the measures are not quantitative and usually are nested. The empirical study is carried out on a sample of 2754 workers of the University of Seville. They have completed a satisfaction questionnaire about training courses aimed to prepare them for the correct performance of their jobs. We highlight the complementarities between both analytical techniques to study the differential variability provided by explained variables nested in different hierarchical level to predict the perceived satisfaction.Downloads
References
Anguera, M .T. (2008). Evaluación de programas desde la metodología cualitativa. Acción Psicológica, 2(5), 87-101.
Anguera, M.T., Chacón-Moscoso, S. y Blanco, A. (2008). Evaluación de programas sociales y sanitarios: Un abordaje metodológico (pp. 173-224). Madrid: Síntesis.
Barbero-García, M. I., Vila-Abad, E. y Holgado-Tello, F.P. (2011). Introducción básica al Análisis Factorial. Madrid: UNED.
Bryk, A. y Raudenbush, S.W. (1992). Hierarchical linear models: Applications and data analysis methods. Newbury Park: Sage Publications.
Campbell, D. T. (1982). Can we be scientific in applied social science? Paper presented at the Annual Meeting of the Educational Research As-sociation. (Reimpreso en R.F. Conner, D.G. Altman y C. Jackson (1984) Evaluation Studies Review Annual, 9, 26-48.
Chacón-Moscoso, S., Sanduvete-Chaves, S., Portell-Vidal, M. & Anguera, M. T. (2013). Reporting a program evaluation: Needs, program plan, intervention, and decisions. International Journal of Clinical and Health Psychology, 13(1), 58-66.
Cook, T. D. (1981). Dilemmas in evaluation of social programs. En M.B. Brewer y B.E. Collins (Eds.), Scientific inquiry and the social sciences. A volume in honor of Donald T. Campbell (pp.257-287). San Francisco: Josey-Bass.
Cook, T. D, Leviton, L. C. y Shadish. W. R. (1985). Program evaluation. En G. Lindzey y E. Aronson (Eds.), Handbook of Social Psychology. (pp.699-777). Nueva York: Random House.
Du Toit, S., Du Toit, M. y Cudeck, R. (1999). Introduction to the analysis of multilevel models with LISREL 8.30. Chicago: Scientific Software International.
Fernández, J. A., y Ovejero, A. (1994). Satisfacción laboral en un centro hospitalario: Un análisis del cuestionario de Porter. Psicología del Trabajo y de las Organizaciones,10, 39–61.
Ferrando, P. y Anguiano, C. (2010). El Análisis Factorial como técnica de investigación en Psicología. Papeles del Psicólogo, 31(1), 18-33.
Gadermann, A. M., Guhn, M. y Zumbo, B. D. (2011). Investigating the substantive aspect of construct validity for the Satisfaction with Life Scale adapted for children: A focus on cognitive processes. Social Indicators Research, 100(1), 37-60.
Herman, J. L., Morris, L. L. y Fitz-Gibbon, C. T. (1987). Program evaluation kit (2ª edición) Beverly Hills: Sage Publications. (9 Vols.).
Holgado-Tello, F. P. y Barbero-García, I. (2008). Innovaciones metodológicas en evaluación psicológica: perspectivas de futuro. Acción Psicológica, 5, 5-6.
Holgado-Tello, F. P., Chacón-Moscoso, S., Barbero-García, I. y Vila-Abad, E. (2010). Polychoric versus Pearson correlations in exploratory and confirmatory factor analysis of ordinal variables. Quality & Quantity, the International Journal of Methodology, 44(1), 153-166.
House, E. R. (1993). Professional Evaluation. Social impact and political conse-quences. Londres: Sage.
House, E. R. (1994). Evaluación, ética y poder. Madrid: Morata.
House, E. R. y Shull, R. D. (1988). Rush to Policy. Using Analytic Techniques in Public Sector Decision-Making. New Brunswick: Transaction Pub-lishers.
Jöreskog, K. G. (2001). Analysis of ordinal variables 2: Cross-Sectional Data. Documentación del Workshop Structural equation modelling with LISREL 8.51. Jena: Friedrich-Schiller-Universität Jena.
Jöreskog, K. G. (2003). Factor Analysis by MINRES. To the memory of Harry Harman and Henry Kaiser. Recuperado de http://www.ssicentral.com/lisrel/techdocs/minres.pdf
Jöreskog, K. G. y Sörbom, D. (1999). PRELIS 2.30. Mooresville, IN: Scientific Software International.
Jöreskog, K. G. y Sörbom, D. (2003). LISREL 8.54. Chicago: Scientific Software International.
Jöreskog, K. G., Sörbom, D., Du Toit, S. y Du Toit, M. (1999). LISREL 8: New statistical features. Chicago: Scientific Software International.
Kirkpatrick, D. (1999). Evaluación de acciones formativas: Los cuatro niveles. Barcelona: Training Club y Epise.
Kreft, I. y Leeuw, J. (1998). Introducing multilevel modeling. Londres: Sage Publications.
Lee, C., Zhang, G. y Edwards, M. C. (2012). Ordinary least squares estimation of parameters in exploratory factor analysis with ordinal data. Multivariate Behavioral Research, 47, 314-339.
Medina, M. (1996). Evaluation of the quality of assistance in social ser-vices. Intervención Psicosocial, 14, 23–42.
Mejías, S. y Huaccho, L. (2011). Macroergonomics intervention pro-grams: recommendations for their design and implementation. Human Factors and Ergonomics in Manufacturing & Service Industries, 21(3), 227–243.
Morata, M. A. y Holgado-Tello, F. P. (2013). Construct validity of Likert scales through Confirmatory Factor Analysis: a simulation study comparing different methods of estimation based on Pearson and polychoric Correlations. International Journal of Social Science Studies, 1(1), 54-61.
Palumbo, D. J. y Nachmias, D. (1983). The preconditions for successful evaluation: is there an ideal paradigm? Policy Sciences, 16, 67-79. (Reimpreso en R. F. Conner, D.G. Altman y C. Jackson (Eds.), Evaluation Studies Review Annual, 9, 102-114).
Passmore, J. y Velez, M. (2012). SOAP-M: A training evaluation model for HR. Industrial and Commer-cial Training, 44(6), 315-325.
Pineda, P. (2010). Evaluation of training in organisations: A proposal for an integrated model. Journal of European Industrial Training, 34(7), 673-693.
Rajeev, P., Madan, M. S. y Jayarajan, K. (2009). Revisiting Kirkpatrick's model - an evaluation of an academic training course. Current Sci-ence, 96(2), 272-276.
Rossi, P. H. y Freeman, H. E. (1985). Evaluation: A systematic approach. Londres: Sage Publications.
Sarkis, J., Gonzalez, P. y Adenso, B. (2010). Stakeholders pressure and the adoption of environmental practices: The mediating effect of training. Journal of Operations Management, 28, 163–176.
Sechrest, L. y Figueredo, A. J. (1993). Program evaluation. Annual Review of Psychology, 44, 645-674.
Thayer, P. W. (1991). A historical perspective on training. En I. L. Goldstein and Associates (Eds.), Training and development in organizations (pp. 457-468). San Francisco: Jossey–Bass.
Ventosa, P. (1998). Desde la evaluación de la formación al rendimiento de la inversión. Barcelona: Epise.
Wilkins, J. L. M. (2010). Modeling quantitative literacy. Educational and Psychological Measurement, 70(2), 267-290.
Yang, F. Jöreskog, K. G. y Luo, H. (2010). Confirmatory Factor Analysis of ordinal variables with misspecified models. Structural Equation Modeling: A Multidisciplinary Journal, 17(3), 392-423.
Yin, H. y Schmeidler, P. J. (2009). Why do standardized ISO 14001 environmental management systems lead to heterogeneous environmental outcomes? Business Strategy and the Environment, 18, 469–486.
The works published in this journal are subject to the following terms:
1. The Publications Service of the University of Murcia (the publisher) retains the property rights (copyright) of published works, and encourages and enables the reuse of the same under the license specified in paragraph 2.
© Servicio de Publicaciones, Universidad de Murcia, 2022
2. The works are published in the online edition of the journal under a Creative Commons Reconocimiento-CompartirIgual 4.0 (legal text). You can copy, use, distribute, transmit and publicly display, provided that: i) you cite the author and the original source of publication (journal, editorial and URL of the work), ii) are not used for commercial purposes, iii ) mentions the existence and specifications of this license.
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
3. Conditions of self-archiving. Is allowed and encouraged the authors to disseminate electronically pre-print versions (version before being evaluated and sent to the journal) and / or post-print (version reviewed and accepted for publication) of their works before publication, as it encourages its earliest circulation and diffusion and thus a possible increase in its citation and scope between the academic community. RoMEO Color: Green.
