Observational data analysis using generalizability theory and general and mixed linear models: an empirical study of infant learning and development

  • Angel Blanco-Villaseñor Universidad de Barcelona
  • Elena Escolano-Pérez Universidad de Zaragoza
Keywords: Systematic observation, General Linear Model, Generalizability Theory, development, learning, childhood.


Accurate evaluation of early childhood competencies is essential for favoring optimal development, as the first years of life form the foundations for later learning and development. Nonetheless, there are still certain limitations and deficiencies related to how infant learning and development are measured. With the aim of helping to overcome some of the difficulties, in this article we describe the potential and advantages of new data analysis techniques for checking the quality of data collected by the systematic observation of infants and assessing variability. Logical and executive activity of 48 children was observed in three ages (18, 21 and 24 months) using a nomothetic, follow-up and multidimensional observational design.

Given the nature of the data analyzed, we provide a detailed methodological and analytical overview of generalizability theory from three perspectives linked to observational methodology: intra- and inter-observer reliability, instrument validity, and sample size estimation, with a particular focus on the participant facet. The aim was to identify the optimal number of facets and levels needed to perform a systematic observational study of very young children.

We also discuss the use of other techniques such as general and mixed linear models to analyze variability of learning and development.

Results show how the use of Generalizability Theory allows controlling the quality of observational data in a global structure integrating reliability, validity and generalizability.


Download data is not yet available.


Albert, J., López-Martín, S., & Carretié, L. (2010). Emotional context modulates response inhibition: Neural and behavioral data. NeuroImage, 49(1), 914-921. doi: 10.1016/j.neuroimage.2009.08.045

Alderson, P. (2004). Ethics. In S. Fraser, V. Lewis, S. Ding, M. Kellett, & C. Robinson (Eds.), Doing research with children and young people (pp. 97-111). London: Sage.

Anguera, M. T. (2001). Cómo apresar las competencias del bebé mediante una aplicación de la metodología observacional. Contextos Educativos, 4, 13-34.

Anguera, M. T. (2010). Posibilidades y relevancia de la observación sistemática por el profesional de la Psicología. Papeles del Psicólogo, 31(1), 122-130. Recuperado de http://www.papelesdelpsicologo.es/pdf/1802.pdf

Anguera, M. T., Blanco-Villaseñor, A., Hernández-Mendo, A., & Losada, J. L. (2011). Diseños observacionales: ajuste y aplicación en psicología del deporte. Cuadernos de Psicología del Deporte, 11(2), 63-76.

Anguera, M. T., Blanco-Villaseñor, A., & Losada, J. L. (2001). Diseños observacionales, cuestión clave en el proceso de la metodología observacional. Metodología de las Ciencias del Comportamiento, 3(2), 135-160.

Aslin, R. N., & Fiser, J. (2005). Methodological challenges for understanding cognitive development in infants. TRENDS in Cognitive Sciences, 9(3), 92-98.

Barkley, R. A. (2012). Executive Functions: What They Are, How They Work, and Why They Evolved. New York: Guilford.

Best, J. R., & Miller, P. H. (2010). A Developmental Perspective on Executive Function. Child Development, 81(6), 1641-1660. doi: 10.1111/j.1467-8624.2010.01499.x

Blanco-Villaseñor, A., Castellano, J., Hernández Mendo, A., Sánchez-López, C. R., & Usabiaga, O. (2014). Aplicación de la TG en el deporte para el estudio de la fiabilidad, validez y estimación de la muestra. Revista de Psicología del Deporte, 23(1), 131-137.

Bornstein, M. H. (2014). Human infancy … and the rest of the lifespan. Annual Review of Psychology, 65, 121-158. doi: 10.1146/annurev-psych-120710-100359

Bryce, D., & Whitebread, D. (2012). The development of metacognitive skills: evidence from observational analysis of young children’s behaviour during problem-solving. Metacognition and Learning, 7(3), 197-217.

Brydges, C. R., Anderson, M., Reid, C. L., & Fox, A. M. (2013). Maturation of cognitive control: delineating response inhibition and interference suppression. PLoS ONE, 8, e69826.

Cardinet, J., Johnson, S., & Pini, G. (2010). Applying Generalizability Theory using EduG. Londres: Routledge.

Carlson, S. M., Faja, S., & Beck, D. M. (2016). Incorporating early development into the measurement of executive function: The need for a continuum of measures across development. In J. A. Griffin, P. McCardle, & L. S. Freund (Eds.), Executive function in preschool-age children: Integrating measurement, neurodevelopment, and translational research (pp. 45-64). Washington, DC, US: American Psychological Association.

Carlson, S. M., Zelazo, P. D., & Faja, S. (2013). Executive function. In P. D. Zelazo (Ed.), The Oxford handbook of developmental psychology, Vol. 1: Body and mind (pp. 706-743). New York: Oxford University Press.

Castellano, J., Blanco-Villaseñor, A., & Álvarez, D. (2011). Contextual variables and time-motion analysis in soccer. International Journal of Sports Medicine, 32(6), 415-421. doi: 0.1055/s-0031-1271771

Clark, A., Flewitt, R., Hammersley, M., & Robb, M. (2013). Understanding Research with Children and Young People. London: Sage.

Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurement: theory of generalizability for scores and profiles. New York: John Wiley and Sons.

Dempster, F. N. (1993). Resistance to interference: Developmental changes in a basic processing mechanism. In M. L. Howe & R. Pasnak (Eds.), Emerging themes in cognitive development (Vol. 1, pp. 3-27). New York, NY: Springer-Verlag.

Dempster, F. N., & Corkill, A. J. (1999). Interference and inhibition in cognition and behavior: unifying themes for education psychology. Educational Psychology Review, 11(2), 1-88.

Diamond, A. (2013). Executive Functions. Annual Review of Psychology, 64, 135-168. doi: 10.1146/annurev-psych-113011-143750

Duan, X., Wei, S., Wang, G., & Shi, J. (2010). The relationship between executive functions and intelligence on 11- to 12-year-old children. Psychological Test and Assessment Modeling, 52, 419-431.

Escolano-Pérez, E., & Blanco-Villaseñor, A. (2015). The longitudinal measurement of change: Intraindividual variability in behavior and interindividual differences observed in childhood. Anales de Psicología, 31(2), 545-551. doi: 10.6018.analesps. 31.2.166361

Escolano-Pérez, E., & Sastre-Riba, S. (2010). Early Infant Cognitive Assessment: Validity of an instrument. Behavior Research Methods, 42(4), 759-767.

Field, M. J., & Behrman, R. E. (Eds.), (2004). Ethical Conduct of Clinical Research Involving Children. Washington (DC): National Academies Press.

Flores-Lázaro, J. C., Castillo Preciado, R. E., & Jiménez-Miramonte, N. A. (2014). Desarrollo de funciones ejecutivas, de la niñez a la juventud. Anales de Psicología, 30(2), 463-473. doi: 10.6018.analesps.30.2.155471

Friedman, N. P., & Miyake, A. (2004). The relations among inhibition and interference control function: a latent-variable analysis. Journal of Experimental Psychology: General, 133(1), 101-135.

Funahashi, S., & Andreau, J. M. (2013). Prefrontal cortex and neural mechanisms of executive function. Journal of Physiology-Paris, 6, 471-482.

García Molina, A., Tirapu Ustárroz, J., & Roig Rovira, T. (2007). Validez ecológica en la exploración de las funciones ejecutivas. Anales de Psicología, 23(2), 289-299.

García, T., Rodríguez, C., González-Castro, P., Álvarez-García, D., & González-Pienda, J. A. (2016). Metacognición y funcionamiento ejecutivo en Educación Primaria. Anales de Psicología, 32(2), 474-483. doi: 10.6018.analesps.32.2.202891

Goswami, U. (2010). The Blackwell Handbook of Childhood Cognitive Development. New York: John Wiley and Sons.

Grupo de Atención Temprana (2000). Libro blanco de la Atención Temprana. Madrid: Real Patronato de Prevención y de Atención a Personas con Minusvalía. Consultado el 28 de Septiembre, 2015, en http://www.gat-atenciontemprana.org/

Guare, R. (2014). Context in the Development of Executive Functions in Children. Applied Neuropsychology Child, 3(3), 226-232. doi: 10.1080/21622965.2013.870015.

Guralnick, M. J. (2015). Merging policy initiatives and developmental perspectives in early intervention. Escritos de Psicología, 8(2), 6-13. doi: 10.5231/psy.writ.2015.1004

Harnishfeger, K. K. (1995). The development of cognitive inhibition: Theories, definitions, and research evidence. In F. N. Dempster & C. J. Brainerd (Eds.), Interference and Inhibition in Cognition (pp. 175-204). San Diego, CA: Academic Press.

Hendry, A., Jones, E. J. H., & Charman, T. (2016). Executive function in the first three years of life: Precursors, predictors and patterns. Developmental Review, 42, 1-33. doi: 10.1016/j.dr.2016.06.005

Herrero, M. L. (1992). Posibilidades de la metodología observacional en el estudio analítico de conductas en el aula: Aplicación en escolares con problemas de comportamiento. Anales de Psicología, 8(1-2), 149-155.

Howard, S. J., Johnson, J., & Pascual-Leone, J. (2014). Clarifying inhibitory control: Diversity and development of attentional inhibition. Cognitive Development, 31(1), 1-21. doi: 10.1016/j.cogdev.2014.03.

Iglesias-Sarmiento, V., Carriedo, N., & Rodríguez, J. L. (2015). Updating executive function and performance in reading comprehension and problem solving. Anales de Psicología, 31(1), 298-309. doi: 10.6018.analesps.31.1.158111

Isquith, P. K., Gioia, G. A., & Espy, K. A. (2004). Executive function in preschool children: examination through everyday behavior. Developmental Neuropsychology, 26(1), 403-422. doi: 10.1207/s15326942dn2601_3

Izard, V., Streri, A., & Spelke, E. S. (2014). Toward Exact number: Young children use one-to-one correspondence to measure set identity but not numerical equality. Cognitive Psychologyst, 72, 27-53. doi: 10.1016./j.cogpschy.2014.01.004

Karmiloff-Smith, A., Casey, B. J., Massand, E., Tomalski, P., & Thomas, M. S. C. (2014). Environmental and genetic influences on neurocognitive development: The importance of multiple methodologies and time-dependent intervention. Clinical Psychological Science, 2, 628-632. doi: 10.1177/2167702614521188

Langer, J. (1986). The origins of logic: One to two years. New York: Academic Press.

Langer, J. (1990). Early cognitive development: Basic funtions. In C.A. Hauert (Ed.), Developmental Psychology: Cognitive, perceptuo-motor and neuro-psychological perspectives (pp. 19-42). Amsterdam: North Holland.

Massand, E., & Karmiloff-Smith, A. (2015). Cascading genetic and environmental effects on development: implications for intervention. In K. Mitchell (Ed.), The Genetics of Neurodevelopmental Disorders (pp. 275-288). Hoboken, U.S.: Wiley-Blackwell.

McArdle, J. J., & Nesselroade, J. R. (2003). Growth curve analysis in contemporary psychological research. In J. Schinka & W. Velicer (Eds.), Comprehensive handbook of psychology: Research methods in psychology (Vol. 2, pp. 447-480). Nueva York: Wiley.

Miranda, A., Colomer, C., Mercader, J., Fernández, I., & Presentación, M. J. (2016). Performance-based tests versus behavioral ratings in the assessment of executive functioning in preschoolers: associations with ADHD symptoms and reading achievement. Frontiers in Psychology, 29. doi: 10.3389/fpsyg.2015.00545

Mishra, J., Anguera, J. A., Ziegler, D. A., & Gazzaley, A. (2014). A Cognitive Framework for Understanding and Improving Interference Resolution in the Brain. Progress in Brain Research, 207, 351-377. doi: 10.1016/B978-0-444-63327-9.00013-8

Mitchell, S. K. (1979). The interobserver agreement, reliability, and generalizability of data collected in observational studies. Psychological Bulletin, 86, 376-390.

Miyake, A., & Friedman, N. P. (2012). The nature and organization of individual differences in executive functions: four general conclusions. Current Directions in Psychological Science, 21(1), 8-14. doi: 10.1177/0963721411429458

Miyake, A., Friedman, N. P., Emerson, M. J., Witzki, A. H., Howerter, A., & Wager, T. D. (2000). The unity and diversity of executive functions and their contributions to complex ‘Frontal Lobe’ tasks: A latent variable analysis. Cognitive Psychology, 41, 49-100.

Moriguchi, Y., Chevalier, N., & Zelazo, P. D. (2016). Editorial: Development of Executive Functions during Childhood. Frontiers in Psychology, 7(6). doi: 10.3389/fpsyg.2016.00006

Muldoon, K., Lewis, C., & Towse, J. (2005). Because it’s there! Why some children count, rather than infer numerical relationships. Cognitive Development, 20, 472-491.

Mushquash, C., & O’Connor, B. P. (2006). SPSS and SAS programs for generalizability theory analyses. Behavior Research Methods, 38, 542-557.

Nee, D. E., & Jonides, J. (2008). Dissociable interference-control processes in perception and memory. Psychological Science, 19(5), 490-500. doi: 10.1111/j.1467-9280.2008.02114.x

Nelson, C. A., & Luciana, M. (2008). Handbook of Developmental Cognitive Neuroscience (2nd Ed.). London: The MIT Press.

Nigg, J. T. (2000). On inhibition/disinhibition in developmental psychopathology: Views from cognitive and personality psychology and a working inhibition taxonomy. Psychological Bulletin, 126(2), 220-246.

Perea, A. E., Alday, L., & Castellano, J. (2006). Registro de datos observacionales a partir del MATCH VISION STUDIO v1.0. In J. Castellano, L. M. Sautu, A. Blanco-Villaseñor, A. Hernández Mendo, A. Goñi, & F. Martínez (Eds.), Socialización y Deporte: Revisión crítica (pp. 135-152). Vitoria-Gasteiz: Arabako Foru Aldundia-Diputación Foral de Álava.

Petersen, I. T., Hoyniak, C. P., McQuillan, M. E., Bates, J. E., & Staples, A. D. (2016). Measuring the development of inhibitory control: The challenge of heterotypic continuity. Developmental Review, 40, 25-71. doi: 10.1016/j.dr.2016.02.001

Salley, B., Panneton, R. K., & Colombo, J. (2013). Separable Attentional Predictors of Language Outcome. Infancy, 18(4), 462-489.

SAS Institute Inc. (2004). SAS 9.1.3 Help and documentation. Cary, NC: SAS Institute Inc.

Scharf, R. J., Scharf, G. J., & Stroustrup, A. (2016). Developmental Milestones. Pediatrics in Review, 37(1), 25-37. doi:10.1542/pir.376266

Schlotzhauer, S. D., & Littell, R. C. (1997). SAS system for elementary statistical analysis. Cary, NC: SAS Institute Inc.

Shanmugan, S., & Satterthwaiter, T. (2015). Neural markers of the development of executive function: relevance for education. Behavioral Sciences, 10, 7-13. doi: 10.1016./j.cobeha.2016.04.007

Shaw, C., Brady, L. M., & Davey, C. (2011). Guidelines for Research with Children and Young People. London: National Children’s Bureau Research Centre.

Sinclair, H., Stambak, M., Lezine, I., Rayna, S., & Verba, M. (1984). Los bebés y las cosas. Barcelona: Gedisa.

Stievano, P., & Valeri, G. (2013). Executive functions in early childhood: Interrelations and structural development of inhibition, set-shifting and working memory. Neuropsychological Trends, 13(1), 27-45. doi: 10.7358/neur-2013-013-stie

Viterbori, P., Usai, M. C., Traverso, L., & De Franchis, V. (2015). How preschool executive functioning predicts several aspects of math achievement in Grades 1 and 3: longitudinal study. Journal Experimental of Child Psychology, 140, 38-55. doi: 10.1016/j.jecp.2015.06.014

Wass, S. V. (2015). Applying cognitive training to target executive functions during early development. Child Neuropsychology, 21(2), 150-166. doi: 10.180/09297049.2014.882888

Wertz, F. J. (2014). Qualitative Inquiry in the History of Psychology. Qualitative Psychology, 1, 4-16.

Whitebread, D., & Coltman, P. (2010). Aspects of pedagogy supporting metacognition and self-regulation in mathematical learning of young children: evidence from an observational study. ZDM Mathematics Education, 42(2), 163-178. doi:10.1007/s11858-009-0233-1

Willoughby, M. T., & Blair, C. B. (2016). Measuring executive function in early childhood: A case for formative measurement. Psychological Assessment, 28(3), 319-330. doi: 10.1037/pas0000152

Willoughby, M. T., Holochwost, S. J., Blanton, Z. E., & Blair, C. B. (2014). Executive Functions: Formative versus Reflective measurement. Measurement: Interdisciplinary Research and Perspectives, 12(3), 69-95. doi: 10.1080/15366367.2014.929.453

Willoughby, M. T., Wirth, R. J., & Blair, C. B. (2011). Contributions of modern measurement theory to measuring executive functions in early childhood: An empirical demostration. Journal of Experimental Child Psychology, 108, 414-435. doi: 10.1016/j.jecp.2010.04.007

Willoughby, M. T., Wirth, R. J., Blair, C. B., & Family Life Project Investigators (2016). Executive Function in Early Childhood: Longitudinal Measurement Invariance and Developmental Change. Psychological Assessment, 28(3), 319-330. doi: 10.1037/pas0000152

Zelazo, P. D., & Carlson, S. M. (2012). Hot and cool executive function in childhood and adolescence: Development and plasticity. Child Development Perspectives, 6(4), 354-360. doi: 10.1111/j.1750-8606.2012.00246.x

How to Cite
Blanco-Villaseñor, A., & Escolano-Pérez, E. (2017). Observational data analysis using generalizability theory and general and mixed linear models: an empirical study of infant learning and development. Anales De Psicología / Annals of Psychology, 33(3), 450-460. https://doi.org/10.6018/analesps.33.3.271021
Monographic theme: Observational data analysis