Design and content validity of a rubric to measure the educational value of applications for mobile devices
Supporting Agencies
- State Research Agency (Ministry of Science and Innovation, Spain).
Abstract
Applications for mobile devices are increasingly a frequently used resource in different educational settings. The objective of this work was the design and validation of an instrument to measure the educational value of these applications, which has been called E-V-A (Educational Value Assessment). A process comprised of three phases was followed, according to a mixed methodology, considering the conceptual and theoretical design of the rubric, the functional design of dimensions, indicators and descriptors, as well as the subsequent validation of theoretical-practical content through expert judgment on relevance, clarity and adequacy, using the Delphi technique. The validation process allowed the formulation of an evaluation rubric for educational apps made up of 100 items structured into 21 blocks and 5 dimensions: personal development, cognitive and metacognitive development, learning and skills development; social development and technical dimension. This instrument can be very useful for teaching professionals who need to measure the educational value of an app through a structured and uniform protocol.
Downloads
References
Abenti, H. F. (2020). How do I teach you? An examination of multiple intelligences and the impact on communication in the classroom. Language & Communication, 73, 29-33. https://doi.org/10.1016/j.langcom.2020.04.001
Almerich, G., Díaz-García, I., Cebrián-Cifuentes, S., & Suárez-Rodríguez, J. (2018). Estructura dimensional de las competencias del siglo XXI en los estudiantes universitarios de educación. RELIEVE - Revista Electrónica de Investigación y Evaluación Educativa, 24(1), Art. 1. https://doi.org/10.7203/relieve.24.1.12548
Arteaga-Martínez, B., Macías-Sánchez, J., Pla-Castells, M., & Ramírez-García, M. (2021). Diseño y validación de un instrumento para observación de clases de matemáticas en Educación Secundaria: Grupo nominal y método Delphi. RELIEVE - Revista Electrónica de Investigación y Evaluación Educativa, 27(2), Art. 2. https://doi.org/10.30827/relieve.v27i2.21812
Berrios-Aguayo, ., Arazola-Ruano, C., & Pantoja-Vallejo, A. (2021). Multiple intelligences: Educational and cognitive development with a guiding focus. South African Journal of Education, 41(2). Scopus. https://doi.org/10.15700/saje.v41n2a1828
Brazuelo-Grund, F., Gallego-Gil, D. J., & Cacheiro-González, M. L. (2017). Los docentes ante la integración educativa del teléfono móvil en el aula. Revista de Educación a Distancia (RED), 52, Art. 52. https://revistas.um.es/red/article/view/282201
Burden, K., & Kearney, M. (2018). Designing an Educator Toolkit for the Mobile Learning Age. International Journal of Mobile and Blended Learning (IJMBL), 10(2), 88-99. https://doi.org/10.4018/IJMBL.2018040108
Cabero-Almenara, J., & Barroso-Osuna, J. (2013). La utilización del juicio de experto para la evaluación de TIC: El Coeficiente de Competencia Experta. Bordón. Revista de Pedagogía, 65(2), Art. 2.
Cabero-Almenara, J., & Llorente-Cejudo, M. C. (2013). La aplicación del juicio de experto como técnica de evaluación de las tecnologías de la información y comunicación (TIC). Eduweb, 7(2), 11-22.
Camilleri, M. A., & Camilleri, A. C. (2019). The students’ readiness to engage with mobile learning apps. Interactive Technology and Smart Education, 17(1), 28-38. https://doi.org/10.1108/ITSE-06-2019-0027
Charter, R. A. (2003). A Breakdown of Reliability Coefficients by Test Type and Reliability Method, and the Clinical Implications of Low Reliability. The Journal of General Psychology, 130(3), 290-304. https://doi.org/10.1080/00221300309601160
Chau, C. L. (2014). Positive Technological Development for Young Children in the Context of Children’s Mobile Apps [Ph.D., Tufts University]. En ProQuest Dissertations and Theses. https://www.proquest.com/docview/1551196695/abstract/91A9D8FA5E03478CPQ/1
Cherner, T. S., Lee, C. H., Fegely, A., & Santaniello, L. A. (2016). A Detailed Rubric for Assessing the Quality of Teacher Resource Apps. Journal of Information Technology Education: Innovations in Practice, 15, 117-143.
Duque, M. P., & Packer, M. J. (2014). Pensamiento y lenguaje. El proyecto de Vygotsky para resolver la crisis de la Psicología. Tesis Psicológica, 9(2), 30-57.
European Commission. (2021). Digital Education Action Plan. https://ec.europa.eu/education/education-in-the-eu/digital-education-action-plan_es
European Parliament. (2016). Directiva (UE) 2016/2102 del Parlamento Europeo y del Consejo, de 26 de octubre de 2016, sobre la accesibilidad de los sitios web y aplicaciones para dispositivos móviles de los organismos del sector público. Official Journal of the European Union, 59, 1-15.
Expósito-López, J. (2018). Acción tutorial en la universidad. Comares.
Expósito-López, J., & Olmedo-Moreno, E. M. (2020). Scientometric analysis of publications on guidance, mentoring, and tutorial action registered on databases. Formación universitaria, 13(3), 123-138. https://doi.org/10.4067/S0718-50062020000300123
Gómez-Domingo, M., & Badia-Garganté, A. (2016). Exploring the use of educational technology in primary education: Teachers’ perception of mobile technology learning impacts and applications’ use in the classroom. Computers in Human Behavior, 56, 21-28. https://doi.org/10.1016/j.chb.2015.11.023
Harrison, R. (2009). Learning and development. Chartered Institute of Personnel and Development.
Hsu, Y. C., & Ching, Y. H. (2015). A review of models and frameworks for designing mobile learning experiences and environments. Canadian Journal of Learning and Technology, 41(3), Art. 3. https://doi.org/10.21432/T2V616
Kalimaya, S., Feranie, S., & Agustin, R. R. (2021). The effect of multiple intelligence theory based teaching towards students’ achievement on electrical circuit topic. 1806(1). Scopus. https://doi.org/10.1088/1742-6596/1806/1/012131
Kearney, M., Burden, K., & Schuck, S. (2019). Disrupting Education Using Smart Mobile Pedagogies. En L. Daniela (Ed.), Didactics of Smart Pedagogy: Smart Pedagogy for Technology Enhanced Learning (pp. 139-157). Springer International Publishing. https://doi.org/10.1007/978-3-030-01551-0_7
Koo, T. K., & Li, M. Y. (2016). A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. Journal of Chiropractic Medicine, 15(2), 155-163. https://doi.org/10.1016/j.jcm.2016.02.012
Krathwohl, D. R., & Anderson, L. W. (2010). Merlin C. Wittrock and the Revision of Bloom’s Taxonomy. Educational Psychologist, 45(1), 64-65. https://doi.org/10.1080/00461520903433562
Lee, C. Y., & Cherner, T. S. (2015). A Comprehensive Evaluation Rubric for Assessing Instructional Apps. Journal of Information Technology Education: Research, 14, 21-53.
Li, K. C., Lee, L. Y.-K., Wong, S.-L., Yau, I. S.-Y., & Wong, B. T.-M. (2018). Effects of mobile apps for nursing students: Learning motivation, social interaction and study performance. Open Learning, 33(2), 99-114. https://doi.org/10.1080/02680513.2018.1454832
Merino-Soto, C., & Livia-Segovia, J. (2009). Intervalos de confianza asimétricos para el índice la validez de contenido: Un programa Visual Basic para la V de Aiken. Anales de Psicología / Annals of Psychology, 25(1), Art. 1.
Real Decreto 1112/2018, de 7 de septiembre, sobre accesibilidad de los sitios web y aplicaciones para dispositivos móviles del sector público, Pub. L. No. Real Decreto 1112/2018, BOE-A-2018-12699 90533 (2018). https://www.boe.es/eli/es/rd/2018/09/07/1112
Morales, J. C., Ramírez, N. E., Vargas, S. H., Peñuela, A. J., Morales, J. C., Ramírez, N. E., Vargas, S. H., & Peñuela, A. J. (2020). Use of Mobile apps in the classroom and its determining factors. Formación universitaria, 13(6), 13-22. https://doi.org/10.4067/S0718-50062020000600013
Oakleaf, M. (2009). Using rubrics to assess information literacy: An examination of methodology and interrater reliability. Journal of the American Society for Information Science and Technology, 60(5), 969-983. https://doi.org/10.1002/asi.21030
Papadakis, S., Kalogiannakis, M., & Zaranis, N. (2017). Designing and creating an educational app rubric for preschool teachers. Education and Information Technologies, 22(6), 3147-3165. https://doi.org/10.1007/s10639-017-9579-0
Penfield, R. D., & Giacobbi, P. R. (2004). Applying a Score Confidence Interval to Aiken’s Item Content-Relevance Index. Measurement in Physical Education and Exercise Science, 8(4), 213-225. https://doi.org/10.1207/s15327841mpee0804_3
Puentedura, R. R. (2014). SAMR and TPCK: A Hands-On Approach to Classroom Practice. Ruben R. Puentedura’s Weblog. Ongoing thoughts on education and technology. http://www.hippasus.com/rrpweblog/archives/000140.html
Salazar-Gómez, E., Tobón, S., & Juárez-Hernández, L. G. (2018). Diseño y validación de una rúbrica de evaluación de las competencias digitales desde la socioformación. Apuntes Universitarios, 8(3), Art. 3. https://doi.org/10.17162/au.v8i3.329
Shoukry, L., Sturm, C., & Galal-Edeen, G. (2015). Pre-MEGa: A Proposed Framework for the Design and Evaluation of Preschoolers’ Mobile Educational Games. The International Conference on Engineering Education, Instructional Technology, Assessment, and E-learning (EIAE 12), Bridgeport: USA. https://doi.org/10.1007/978-3-319-06773-5_52
Stoyanov, S. R., Hides, L., Kavanagh, D. J., Zelenko, O., Tjondronegoro, D., & Mani, M. (2015). Mobile App Rating Scale: A New Tool for Assessing the Quality of Health Mobile Apps. JMIR MHealth and UHealth, 3(1), e3422. https://doi.org/10.2196/mhealth.3422
Sweeney, P., & Moore, C. (2012). Mobile Apps for Learning Vocabulary: Categories, Evaluation and Design Criteria for Teachers and Developers. International Journal of Computer-Assisted Language Learning and Teaching (IJCALLT), 2(4), 1-16. https://doi.org/10.4018/ijcallt.2012100101
Symeonaki, M., Filandrianos, G., & Stamou, G. (2022). Visualising key information and communication technologies (ICT) indicators for children and young individuals in Europe. Humanities and Social Sciences Communications, 9(1). Scopus. https://doi.org/10.1057/s41599-022-01356-5
Tabuenca, B., Sánchez-Peña, J. J., & Cuetos-Revuelta, M. J. (2019). El smartphone desde la perspectiva docente: ¿una herramienta de tutorización o un catalizador de ciberacoso? Revista de Educación a Distancia (RED), 19(59), Art. 59. https://doi.org/10.6018/red/59/01
Taylor, G., Kolak, J., Bent, E. M., & Monaghan, P. (2022). Selecting educational apps for preschool children: How useful are website app rating systems? British Journal of Educational Technology, 53(5), 1262-1282. https://doi.org/10.1111/bjet.13199
UNESCO. (2019). Tesauro de la UNESCO (Educación). https://vocabularies.unesco.org/browser/thesaurus/es/page/domain1
Wai, I. S. H., Ng, S. S. Y., Chiu, D. K. W., Ho, K. K. W., & Lo, P. (2018). Exploring undergraduate students’ usage pattern of mobile apps for education. Journal of Librarianship and Information Science, 50(1), 34-47. https://doi.org/10.1177/0961000616662699
Walker, H. (2011). Evaluating the effectiveness of apps for mobile devices. Journal of Special Education Technology, 26(4), 59-63. https://doi.org/10.1177/016264341102600405
Weng, P. L. (2015). Developing an App Evaluation Rubric for Practitioners in Special Education. Journal of Special Education Technology, 30(1), 43-58. https://doi.org/10.1177/016264341503000104
Zapata-Ros, M. (2012). Calidad en entornos ubicuos de aprendizaje. Revista de Educación a Distancia (RED), 31, Art. 31. https://revistas.um.es/red/article/view/232871
Copyright (c) 2023 Distance Education Journal
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Las obras que se publican en esta revista están sujetas a los siguientes términos:
1. El Servicio de Publicaciones de la Universidad de Murcia (la editorial) conserva los derechos patrimoniales (copyright) de las obras publicadas, y favorece y permite la reutilización de las mismas bajo la licencia de uso indicada en el punto 2.
2. Las obras se publican en la edición electrónica de la revista bajo una licencia Creative Commons Reconocimiento-NoComercial-SinObraDerivada 3.0 España (texto legal). Se pueden copiar, usar, difundir, transmitir y exponer públicamente, siempre que: i) se cite la autoría y la fuente original de su publicación (revista, editorial y URL de la obra); ii) no se usen para fines comerciales; iii) se mencione la existencia y especificaciones de esta licencia de uso.
3. Condiciones de auto-archivo. Se permite y se anima a los autores a difundir electrónicamente las versiones pre-print (versión antes de ser evaluada) y/o post-print (versión evaluada y aceptada para su publicación) de sus obras antes de su publicación, ya que favorece su circulación y difusión más temprana y con ello un posible aumento en su citación y alcance entre la comunidad académica. Color RoMEO: verde.
