Quality Measurement in E-Learning for Higher Education: Lack of Consensus and the Need for Standardization
Supporting Agencies
- Universidad Santo Tomás
Abstract
The growth of e-learning systems in higher education institutions worldwide has underscored the need for a rigorous and systematic assessment of their quality. This study aims to analyze the current state of quality measurement in e-learning systems through a systematic literature review. The SPIDER strategy was applied within the PRISMA protocol framework, utilizing high-impact academic databases, including SCOPUS, Web of Science, EBSCOhost, Dialnet, JSTOR, Science Direct, and ProQuest. Articles published between 1992 and 2024 were collected, and after applying eligibility criteria and quality filters, 66 studies were selected for analysis. The results reveal a wide range of models, standards, and guidelines for evaluating e-learning quality, most of which lack a standardized measurement method. Furthermore, among the studies that applied a quality measurement approach, there was a lack of consensus in the metrics used, making cross-comparison of findings unfeasible. Notably, none of the reviewed studies validated their results against a globally recognized standard, indicating the absence of a unified framework for e-learning quality assessment. Given this scenario, we recommend the formation of an international technical committee composed of standardization bodies and academic institutions to develop a consensus-based measurement method and establish a global standard for ensuring the quality of e-learning systems.
Downloads
-
Abstract119
-
PDF (Español (España))76
References
Adom, D., Adu-Mensah, J., & Dake, D. A. (2020). Test, measurement, and evaluation: Understanding and use of the concepts in education. International Journal of Evaluation and Research in Education (IJERE), 9(1), 109. https://doi.org/10.11591/ijere.v9i1.20457
Alizadeh, M., Mehran, P., Koguchi, I., & Takemura, H. (2019). Evaluating a blended course for Japanese learners of English: Why Quality Matters: Revista de Universidad y Sociedad del Conocimiento: Revista de Universidad y Sociedad del Conocimiento. International Journal of Educational Technology in Higher Education, 16(1), 1-21. ProQuest One Academic. https://doi.org/10.1186/s41239-019-0137-2
Alkhattabi, M., Neagu, D., & Cullen, A. (2011). Assessing information quality of e-learning systems: A web mining approach. Web 2.0 in Travel and Tourism: Empowering and Changing the Role of Travelers, 27(2), 862-873. https://doi.org/10.1016/j.chb.2010.11.011
Amaliah Nadir, R. D., Athaya, H., Sensuse, D. I., Kautsarina, & Suryono, R. R. (2021). Factors Influencing E-learning System Success during COVID-19 Pandemic (Case Study: Faculty of Computer Science, Universitas Indonesia). 2021 International Conference on Advanced Computer Science and Information Systems, ICACSIS 2021. Scopus. https://doi.org/10.1109/ICACSIS53237.2021.9631313
Baldomero, M., Vázquez Cano, E., & Belando Montoro, M. R. (2018). Diseño de un modelo de evaluación de la calidad de los cursos MOOC mediante Lógica difusa. REDIE: Revista Electrónica de Investigación Educativa, 20(4), 72-85. https://0-dialnet-unirioja-es.llull.uib.es/servlet/extart?codigo=6716945
Balyk, N., Oleksiuk, V., & Shmyger, G. (2017). Development of e-Learning quality assessment model in Pedagogical University. CEUR Workshop Proceedings, 1844, 440-450. Scopus. https://www.scopus.com/inward/record.uri?eid=2-s2.0-85020533433&partnerID=40&md5=8f7ec0ec12b1ff11b62e9da06314761d
Beiqing, C., & Chunrong, Z. (2020). Design of online teaching quality evaluation system for Private University: - Research based on deep learning algorithm. Proceedings - 2020 International Conference on Modern Education and Information Management, ICMEIM 2020, 29-32. Scopus. https://doi.org/10.1109/ICMEIM51375.2020.00015
Béjar, M. de la O. C., & Vera, M. del M. S. (2022). Cambio de modalidad presencial a virtual durante el confinamiento por Covid-19: Percepciones del alumnado universitario. Revista Iberoamericana de Educación a Distancia, 25(1), 243-260. Coronavirus Research Database; ProQuest One Academic. https://doi.org/10.5944/ried.25.1.30623
Berra, S. (2020). Fundamentos y Método de las Revisiones Sistemáticas. Areté, 20(2), Article 2. https://doi.org/10.33881/1657-2513.art.20208
Booth, A. (2006). Clear and present questions: Formulating questions for evidence based practice. Library Hi Tech, 24(3), 355-368. https://doi.org/10.1108/07378830610692127
Cabero Almenara, J., Del Prete, A., & Arancibia Muñoz, M. L. (2020). Modelo para determinar acciones de calidad en la formación virtual. Digital Education Review, 37, 323-342. https://idus.us.es/bitstream/handle/11441/65798/guia%20para%20la%20evaluacion%20didactica%20de%20cursos%20de%20teleformacion.pdf?sequence=1
Campbell, M., McKenzie, J. E., Sowden, A., Katikireddi, S. V., Brennan, S. E., Ellis, S., Hartmann-Boyce, J., Ryan, R., Shepperd, S., Thomas, J., Welch, V., & Thomson, H. (2020). Synthesis without meta-analysis (SWiM) in systematic reviews: Reporting guideline. BMJ, 368, l6890. https://doi.org/10.1136/bmj.l6890
Carter, J. (1994). Book Reviews: Evaluation Thesaurus (4th ed.), by Michael Scriven. Newbury Park: Sage Publications, 1991, 391 pp. Evaluation Practice, 15(1), 109-110. https://doi.org/10.1177/109821409401500117
Castillo, R. Q. (s. f.). CONCEPTOS BÁSICOS DE LA EVALUACIÓN DEL APRENDIZAJE. https://www.iisue.unam.mx/perfiles/descargas/pdf/1988-41-42-48-51
Chiyon, I., Quevedo, A. V., Vegas, S., & Mosquera, J. C. (2021). An evaluation method of the impact of an online teaching system on engineering students’ satisfaction during the COVID-19 lockdown. Proceedings - 7th International Symposium on Accreditation of Engineering and Computing Education, ICACIT 2021. Scopus. https://doi.org/10.1109/ICACIT53544.2021.9612504
Choi, C.-R., & Jeong, H.-Y. (2019). Quality evaluation for multimedia contents of e-learning systems using the ANP approach on high speed network. Multimedia Tools and Applications, 78(20), 28853-28875. Scopus. https://doi.org/10.1007/s11042-019-7351-8
Cooke, A., Smith, D., & Booth, A. (2012). Beyond PICO: The SPIDER tool for qualitative evidence synthesis. Qualitative Health Research, 22(10), 1435-1443. https://doi.org/10.1177/1049732312452938
Das, G. (2014), “Linkages of retailer awareness, retailer association, retailer perceived quality and retailer loyalty with purchase intention: a study of Indian food retail brands”. Journal of Retailing and Consumer Services, 21 (3), pp. 284-292. https://www.sciencedirect.com/science/article/abs/pii/S0969698913001124
Denyer, D., Tranfield, D., & Aken, van. (2008). Developing design propositions through research synthesis. Organization Studies, 29(3), 393-413. https://doi.org/10.1177/0170840607088020
Edward, M. y Sahadev, S. (2011), “Role of switching costs in the service quality, perceived value, customer satisfaction and customer retention linkage”. Asia Pacific Journal of Marketing and Logistics, 23 (3), pp. 327-345. https://doi.org/10.1108/13555851111143240
Escobio-Prieto, I. et al. (2021) Analysis of the satisfaction degree of students at Spain’s physiotherapy universities in relation to online teaching during the COVID-19 pandemic. Sustainability 2021, 13, 13628. https:// doi.org/10.3390/su132413628
Esmaeili Givi, M., Keshavarz, H., & Kargar Azad, Z. (2022). Quality assessment of E-learning website using asymmetric impact–performance analysis and Kano’s customer satisfaction model: A case study based on WebQual 4.0. Information Discovery and Delivery. https://doi.org/10.1108/IDD-08-2021-0083
Garde, R. A., & González-Torres, M. C. (2021). Educación de calidad y reconstrucción resiliente en el alumnado. Estudio piloto durante el confinamiento por la Covid-19. DEDiCA Revista de Educação e Humanidades (dreh), 18, Art. 18. https://doi.org/10.30827/dreh.vi18.17993
Gartlehner, G., Affengruber, L., Titscher, V., Noel-Storr, A., Dooley, G., Ballarini, N., & König, F. (2020). Single-reviewer abstract screening missed 13 percent of relevant studies: A crowd-based, randomized controlled trial. Journal of Clinical Epidemiology, 121, 20-28. https://doi.org/10.1016/j.jclinepi.2020.01.005
Hay, D. B., Kehoe, C., Miquel, M. E., Hatzipanagos, S., Kinchin, I. M., Keevil, S. F., & Lygo‐Baker, S. (2008). Measuring the quality of e-learning. British Journal of Educational Technology, 39(6), 1037-1056. Academic Search Ultimate. https://search.ebscohost.com/login.aspx?direct=true&db=asn&AN=34805549&lang=es&site=ehost-live
Hamer, L. O. A. (2006), “Confirmation perspective on perceived service quality”. Journal of Services Marketing, 20 (4), pp. 219-232. https://doi.org/10.1108/08876040610674571
Harasim, L. (2006). A History of E-learning: Shift Happened. In: Weiss, J., Nolan, J., Hunsinger, J., Trifonas, P. (eds) The International Handbook of Virtual Learning Environments. Springer, Dordrecht. https://doi.org/10.1007/978-1-4020-3803-7_2
Hou, J. (2023). The method of online classroom teaching quality evaluation based on deep data mining. International Journal of Continuing Engineering Education and Life-Long Learning (IJCEELL), 33(4). https://doi.org/10.1504/ijceell.2023.132388
ISO 3534-2:2006(en), Statistics—Vocabulary and symbols—Part 2: Applied statistics. (s. f.). Recuperado 23 de febrero de 2023, de https://www.iso.org/obp/ui/#iso:std:iso:3534:-2:ed-2:v1:en
ISO 9000:2015(es), Sistemas de gestión de la calidad—Fundamentos y vocabulario. (s. f.). Recuperado 23 de febrero de 2023, de https://www.iso.org/obp/ui/es/#iso:std:iso:9000:ed-4:v1:es
Jones, A., Scanlon, E., Tosunoglu, C., Morris, E., Ross, S., Butcher, P., & Greenberg, J. (1999). Contexts for evaluating educational software. Interacting with Computers, 11(5), 499-516. https://doi.org/10.1016/S0953-5438(98)00064-2
Laurillard, D. (2002). Rethinking university teaching: A conversational framework for the effective use of learning technologies (2nd ed.). London: Routledge Falmer. doi:10.4324/9780203304846
Li, W. (2023). A Model for Assessing the Qquality of a Distance Education Programme in an Online Environment: China’s Experience. Croatian Journal of Education, 213-246.
Li, W. (2024). A construction of online teaching quality evaluation model based on big data mining. International Journal of Continuing Engineering Education and Life-Long Learning (IJCEELL), 34(1). https:// doi.org/ 10.1504/ijceell.2024.135271
Lint, M., Wang, H., & Xu, Z. (2019). TODIM-based multi-criteria decision-making method with hesitant fuzzy linguistic term sets. Artificial Intelligence Review. https://www.semanticscholar.org/paper/TODIM-based-multi-criteria-decision-making-method-Lin-Wang/d56ba2b4ad22aac2ba23531db40db76a55fa0061
Linn, R. L., & Gronlund, N. E. (2000). Measurement and Assessment in Teaching. Eighth Edition. Prentice-Hall, Order Processing Center, P.
Liu, L., Liu, X., & Xu, F. (2021). The study of quality evaluation model for the real-time interactive online teaching. 2021 10th International Conference on Educational and Information Technology, ICEIT 2021, 61-65. Scopus. https://doi.org/10.1109/ICEIT51700.2021.9375621
Lynch, B. K. (2001). Rethinking assessment from a critical perspective. Language Testing, 18(4), 351-372. https://doi.org/10.1177/026553220101800403
Martínez-Caro, E., Cegarra-Navarro, J. G., & Cepeda-Carrión, G. (2015). An application of the performance-evaluation model for e-learning quality in higher education. Total Quality Management & Business Excellence, 26(5-6), 632. ProQuest One Academic. https://www.proquest.com/scholarly-journals/application-performance-evaluation-model-e/docview/1669844557/se-2?accountid=29068
McKenzie, J. E., & Brennan, S. E. (2019). Synthesizing and presenting findings using other methods. En Cochrane Handbook for Systematic Reviews of Interventions (pp. 321-347). John Wiley & Sons, Ltd. https://doi.org/10.1002/9781119536604.ch12
Methley, A. M., Campbell, S., Chew-Graham, C., McNally, R., & Cheraghi-Sohi, S. (2014). PICO, PICOS and SPIDER: A comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Services Research, 14, 579. https://doi.org/10.1186/s12913-014-0579-0
Moliner, M. A. (2009), “Loyalty, perceived value and relationship quality in healthcare services”. Journal of Service Management, 20 (1), pp. 76-97. https://doi.org/10.1108/09564230910936869
Muhammad, A. H., Siddique, A., Youssef, A. E., Saleem, K., Basit Shahzad, Akram, A., & Al-Batool Saleh Al-Thnian. (2020). A Hierarchical Model to Evaluate the Quality of Web-Based E-Learning Systems. Sustainability, 12(10), 4071. Coronavirus Research Database; ProQuest One Academic. https://doi.org/10.3390/su12104071
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015). Using text mining for study identification in systematic reviews: A systematic review of current approaches. Systematic Reviews, 4(1), 5. https://doi.org/10.1186/2046-4053-4-5
Quality Matters. (s. f.). Recuperado 18 de abril de 2023, de https://www.qualitymatters.org/qa-resources/rubric-standards/higher-ed-rubric
Ramírez-Fernández, M. B., & Silvera, J. L. S. (2015). EDUTOOL®: UN INSTRUMENTO PARA LA EVALUACIÓN Y ACREDITACIÓN DE LA CALIDAD DE LOS MOOCS. Educación XX. https://www.redalyc.org/pdf/706/70638708004.pdf
Richardson, W., Wilson, M., Nishikawa, J., & Hayward, R. (1995). The well-built clinical question: A key to evidence-based decisions. ACP Journal Club. https://www.semanticscholar.org/paper/The-well-built-clinical-question%3A-a-key-to-Richardson-Wilson/9997b4d1eb83cfeb23c673f5643e3b9bf545763e
Sáiz Manzanares, M. C., Casanova, J., Lencastre, J. A., Almeida, L. S., & Martín Antón, L. J. (2022). Satisfacción de los estudiantes con la docencia online en tiempos de COVID-19. Comunicar: Revista científica iberoamericana de comunicación y educación, 70, 35-45. https://0-dialnet-unirioja-es.llull.uib.es/servlet/extart?codigo=8198767
Saleh, A. M., Abuaddous, H. Y., Alansari, I. S., & Enaizan, O. (2022). The Evaluation of User Experience of Learning Management Systems Using UEQ. International Journal of Emerging Technologies in Learning, 17(7), 145-162. Scopus. https://doi.org/10.3991/ijet.v17i07.29525
Scriven, M. (1991). Evaluation thesaurus, 4th ed (pp. xiii, 391). Sage Publications, Inc.
Shemilt, I., Khan, N., Park, S., & Thomas, J. (2016). Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews. Systematic Reviews, 5(1), 140. https://doi.org/10.1186/s13643-016-0315-4
Snoj, B. - Korda, A. P. y Mumel, D. (2004), “The relationships among perceived quality, perceived risk and perceived product value”. Journal of Product & Brand Management, 13 (3), pp. 156-167. https://doi.org/10.1108/10610420410538050
Su, B. (2021). Research of Online Course Quality Evaluation System by AHP Entropy Algorithm and Weight Coefficient. 2021 IEEE International Conference on Advances in Electrical Engineering and Computer Applications (AEECA), 655-660. https://doi.org/10.1109/AEECA52519.2021.9574212
Sujono, & Santoso, H. B. (2017). E-Learning Quality Analysis Of Use Of Web Conference In The Improvement Of Students With Learning Method Webqual (Case Study: Universitas KH. A. Wahab Hasbullah). IEESE International Journal of Science and Technology, 6(1), 8-14. ProQuest One Academic. https://www.proquest.com/scholarly-journals/e-learning-quality-analysis-use-web-conference/docview/1890051876/se-2?accountid=29068
Tang, B., Guo, S., Yeboah, M., Wang, Z., & Cheng, S. (2021). Quality evaluation of online courses during COVID-19 pandemic based on integrated FCE-AHP method. Journal of Intelligent and Fuzzy Systems, 41(1), 1487-1498. Scopus. https://doi.org/10.3233/JIFS-210362
Theresiawati, T., Seta, H., & Arista, A. (2023). Implementing quality function deployment using service quality and Kano model to the quality of e-learning. International Journal of Evaluation and Research in Education (IJERE), 12, 1560. https://doi.org/10.11591/ijere.v12i3.25511
Tobías Martínez, M. Á., Fuentes Esparrell, J. A., Feriche Rodríguez, I., & Alvarez Jiménez, F. J. (2020). Evaluación de la calidad en servicios de educación superior a distancia: Escala SERVQUAL y análisis factorial. Etic@net: Revista científica electrónica de Educación y Comunicación en la Sociedad del Conocimiento, 20(2), 306-335. https://0-dialnet-unirioja-es.llull.uib.es/servlet/extart?codigo=7656736
Torres Barzabal, L., Barcia, D. M., & Ortiz, P. (2017). Assessment of online learning at the Andalusian virtual campus. The case of Pablo Olavide University subjects. InterCambios: Dilemas y Transiciones de la Educación Superior, 4(1), 48-57. https://0-dialnet-unirioja-es.llull.uib.es/servlet/extart?codigo=6064346
Torres-Díaz, J. C., Rivera-Rogel, D., Beltrán-Flandoli, A. M., & Andrade-Vargas, L. (2022). Effects of COVID-19 on the Perception of Virtual Education in University Students in Ecuador; Technical and Methodological Principles at the Universidad Técnica Particular de Loja. Sustainability, 14(6), Art. 6. https://doi.org/10.3390/su14063204
Tripathi, M. & V.K.J. Jeevan. (2009). Quality assurance in distance learning libraries. Quality Assurance in Education, 17(1), 45-60. ProQuest One Academic. https://doi.org/10.1108/09684880910929926
Vila, R. R., Mengual-Andrés, S., & Guerrero, C. S. (2014). Assessment the pedagogical quality of the MOOC. Profesorado, 18(1), 27-41. Scopus. https://www.scopus.com/inward/record.uri?eid=2-s2.0-84903763155&partnerID=40&md5=c6e6e9e2b0a36f506df7c4cde2dd4773
Waffenschmidt, S., Knelangen, M., Sieben, W., Bühn, S., & Pieper, D. (2019). Single screening versus conventional double screening for study selection in systematic reviews: A methodological systematic review. BMC Medical Research Methodology, 19(1), 132. https://doi.org/10.1186/s12874-019-0782-0.
Wang, Z., Nayfeh, T., Tetzlaff, J., O’Blenis, P., & Murad, M. H. (2020). Error rates of human reviewers during abstract screening in systematic reviews. PLOS ONE, 15(1), e0227742. https://doi.org/10.1371/journal.pone.0227742
Wildridge, V. (s. f.). How CLIP became ECLIPSE: A mnemonic to assist in searching for health policy/management information. Health Information and Libraries Journal, 19(2), 113-115. http://resolver.scholarsportal.info/resolve/14711834/v19i0002/113_hcbeamisfhpi.xml
Zhang, Y., & Liu, C. (2023). Online Teaching Quality Evaluation: Entropy TOPSIS and Grouped Regression Model. International Journal of Emerging Technologies in Learning (iJET), 18(16), Article 16. https://doi.org/10.3991/ijet.v18i16.41353
Zhou, Y., & Zhang, L. (2023). 1. Evaluation method of online education quality based on fuzzy rough set. International Journal of Continuing Engineering Education and Life-Long Learning (IJCEELL), 19(4). https:// doi.org/10.1504/ijceell.2023.132419
Copyright (c) 2025 frank Delgado
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Those authors who have publications with this journal accept the following terms:
a. The authors will retain their copyright and guarantee the journal the right of first publication of their work, which will be simultaneously subject to the Creative Commons License. Non-commercial attribution 4.0 International that allows to share, copy, and redistribute the material in any medium or format and adapt, remix, transform and build on the material in the following terms:
Recognition - You must give the appropriate credit, provide a link to the license, and indicate if changes have been made. You may do so in any reasonable manner, but not in a way that suggests that the licensor or its use endorses it. Non-commercial - You cannot use the material for commercial purposes. Share under it - If you remix, transform, or create on the material, your contributions must be distributed under the same license as the original.
b. Authors may adopt other non-exclusive licensing agreements for the distribution of the published work (e.g. deposit it in an institutional telematic file or publish it in a monographic volume) whenever the initial publication in this journal is indicated.
c. Authors are allowed and encouraged to distribute their work through the Internet (e.g. in institutional telematic archives or on their website) before and during the submission process, which can produce interesting exchanges and increase citations of the published work. (See The effect of open access).
d. In any case, the Editorial Team understands that the opinions expressed by the authors are their exclusive responsibility.
