JOLer: A Java standalone application for simulating the Weaver & Kelemen's judgment of learning (JOL) model.

  • Marcos Ruiz UNED
  • Cristobal Arroyo UNED
Keywords: metamemory, metacognition, judgment of learning, JOL, gamma correlation, simulation, Java,


To assess judgment of learning (JOL) accuracy in metamemory, researchers have to measure how much the metamemory judgments adjust to the participant's memory-test performance. Absolute accuracy or calibration is the average correspondence between JOL and memory performance. Metamemory relative accuracy or resolution is a measure of how sensitive a participant is to the differential recallability between two studied items. Unfortunately, factors altering both calibration and resolution very often change also the distribution of JOL on the available scale for judgment. The problem with these effects on JOL distribution is that they could yield an altered resolution estimation due to the way in which its usual estimate is computed. JOLer simulates the behavior of participants in a typical metamemory procedure. The application is offered as a tool for metamemory researchers: it affords the opportunity to check whether, maintaining calibration parameters but changing JOL distributions between conditions, a different (and somewhat spurious) resolution estimate would be obtained.


Arbuckle, T. Y. & Cuddy, L.L. (1969). Discrimination of item strength at time of presentation. Journal of Experimental Psichology, 81, 126-131.

Arnold, M.M., Higham, P.A., & Martín-Luengo, B. (2013). A little bias goes a long way: The effects of feedback on the strategic regulation of accuracy on formula-scored tests. Journal of Experimental Psychology: Applied, 19, 383-402.

Begg, I., Duft, S., Lalonde, P. Melnick, R., & Sanvito, J. (1989). Memory predictions are based on ease of processing. Journal of Memory & Language, 28, 610-632.

Benjamin, A.S. & Díaz, M. (2008). Measurement of relative metamnemonic accuracy. In J. Dunlosky & R.A. Bjork (Eds.), Handbook of memory and metamemory (pp. 73-94). New York, NY: Psychology Press.

Brown, R & Mcneill, D. (1966). Te “tip of the tongue” phenomenon. Journal of Verbal Learning & Behavior, 5, 325-337.

Dunlosky, J. & Bjork, R.A. (Eds.) (2008). Handbook of Metamemory and Memory. Hove, NJ: Psychology Press.

Dunlosky, J. & Metcalfe, J. (2009). Metacognition. Thousand Oaks, Ca: Sage.

Dunlosky, J. & Nelson, T.O. (1992). Importance of the kind of cue for judgments of learning (JOL) and the delayed-JOL effect. Memory & Cognition, 20, 373-380.

Dunlosky, J. & Nelson, T.O. (1994). Does the sensitivity of judgments of learning (JOLs) to the effect of various activities depend on when the JOLs occur? Journal of Memory and Language, 33, 545-565.

Finn, B. & Metcalfe, J. (2007). The role of memory for past test in the underconfidence with practice effect. Journal of Experimental Psychology, Learning, Memory, and Cognition, 33, 238-244.

Finn, B. & Metcalfe, J. (2014). Overconfidence in children's multitrial judgments of learning. Learning and Instruction, 32, 1-9.

Glenberg, A.M. & Epstein, W. (1987). Inexpert calibration of comprehension. Memory & Cognition, 15, 84–93.

Glucksberg, S. & McCloskey, M. (1981). Decisions about ignorance: Knowing that you don't know. Journal of Experimental Psychology: Human Learning and Memory, 7, 311-325.

González, R. & Nelson, T.O. (1996). Measuring ordinal association in situations that contain tied scores. Psychological Bulletin, 119, 159-165.

Goodman, L.A. & Kruskal, W.H. (1954). Measures of association for cross classifications. Journal of the American Statistical Association, 49, 732-764.

Hart, J.T. (1965). Memory and the feeling of knowing experience. Journal of Educational Psichology, 56, 208-216.

Hart, J.T. (1967). Memory and the memory-monitoring process. Journal of Verbal Learning and Verbal Behavior, 6, 685-691.

Kelley, C. M. & Jacoby, L. L. (1996). Memory attributions: Remembering, knowing, and feeling of knowing. In L. M. Reder (Ed.), Implicit memory and metacognition (pp. 287-307). Mahwah, NJ: Erlbaum.

Koriat, A. (1993). How do we know that we know? The accessibility model of the feeling of knowing. Psychological Review, 100, 609-639.

Koriat, A. (1997). Monitoring one’s own knowledge during study: a cue utilization approach to judgments of learning. Journal of Experimental Psichology: General, 126, 349-370.

Koriat, A. & Goldsmith, M. (1996). Monitoring and control processes in the strategic regulation of memory accuracy. Psychological Review, 103, 490-517.

Koriat, A. & Levy-Sadot, R. (2001). The Combined Contributions of the Cue-Familiarity and Accessibility Heuristics to Feelings of Knowing. Journal of Experimental Psychology: Learning, Memory, and Cognition, 27, 34-53.

Koriat, A., Ma’ayan, H., Sheffer, L., & Bjork, R. A. (2006). Exploring a mnemonic debiasing account of the underconfidence-with-practice effect. Journal of Experimental Psychology: Learning, Memory, and Cognition, 32, 595-608.

Koriat, A. & Nussinson, R. (2009). Attributing study effort to data-driven and goal-driven effects: Implications for metacognitive judgments. Journal of Experimental Psychology: Learning, Memory, and Cognition, 35, 338–1343.

Koriat, A., Sheffer, L., & Ma’ayan, H. (2002). Comparing objective and subjective learning curves: Judgment of learning exhibit increased underconfidence-with-practice. Journal of Experimental Psychology: General, 131, 147-162.

Liberman, V. & Tversky, A. (1993). On the evaluation of probability judgments: Calibration, discrimination, and monotonicity. Psychological Bulletin, 114, 162-173.

Lovelace, E. A. (1984). Metamemory: Monitoring Future Recallability During Study. Journal of Experimental Psychology: Learning, Memory, and Cognition, 10, 756-766.

Luna, K., Higham, P.A., & Martín-Luengo, B. (2011). Regulation of memory accuracy with multiple answers: The plurality option. Journal of Experimental Psychology: Applied, 17, 148-158.

Masson, M.E.J. & Rotello, C.M. (2009). Sources of bias in the Goodman-Kruskal gamma coefficient measure of associations: Implications for studies of metacognitive processes. Journal of Experimental Psychology: Learning, Memory, and Cognition, 35, 509-527.

Metcalfe, J. & Finn, B. (2008). Familiarity and Retrieval Processes in Delayed Judgments of Learning. Journal of Experimental Psychology: Learning, Memory, and Cognition, 34, 1084–1097.

Metcalfe., J., Schwartz, B. L. & Joaquim, S. G. (1993). The cue-familiarity heuristic in metacognition. Journal of Experimental Psychology: Learning, Memory, and Cognition, 19, 851-861.

Muruyama, K., Sakaki, M., Yan, V.X., & Smith, G.M. (2014). Type I error inflation in the traditional by-participant analysis to metamemory accuracy: A generalized mixed-effects model perspective. Journal of Experimental Psychology: Learning, Memory, and Cognition, 40, 1287-1306.

Nelson, T.O. (1984). A comparison of current measures of the accuracy of feeling-of-knowing predictions. Psychological Bulletin, 95, 109-133.

Nelson, T.O. y Dunlosky, J. (1991). When people’s judgments of learning (JOLs) are extremely accurate at predicting recall: The “delayed-JOL effect”. Psychological Science, 2, 267-270.

Nelson. T.O., Leonesio, R.J ., Landwehr. R.S., & Narens. L. (1986). A comparison of three predictors of an individual's memory performance: The individual's feeling of knowing versus the normative feeling of knowing versus base-rate item difficulty. Journal of Experimental Psychology: Learning, Memory, and Cognition, 12, 279-287.

Nelson, T. O. & Narens, L. (1980). Norms of 300 general-information questions: Accuracy of recall, latency of recall, and felling-ofknowing ratings. Journal of Verbal Learning and Verbal Behavior, 19, 338–368.

Nelson, T.O., y Narens, L. (1990). Metamemory: A theoretical framework and new findings. The Psychology of Learning and Motivation, 26, 125-173.

Pyc, M.A., Rawson, K.A., & Aschenbrenner, A.J. (2014). Metacognitive monitoring during criterion learning: When and why are judgments accurate. Memory & Cognition, 42, 886–897.

Reder, L.M. (Eds.) (1996). Implicit Memory and Metacognition. Mahwah, NJ: Lawrence Erlbaum.

Rhodes, M. G. & Tauber, S.K. (2011). The Influence of Delaying Judgments of Learning on Metacognitive Accuracy: A Meta-Analytic Review. Psychological Bulletin, 137, 131–148.

Ruiz, M. (2004). Las caras de la memoria. Madrid: Pearson-Prentice Hall.

Serra, M.J. & Ariel, R. (2014). People use the memory for past-test heuristic as an explicit cue for judgments of learning. Memory & Cognition, 42, 1260–1272.

Shanks, L.I. & Serra, M.J. (2014). Domain familiarity as a cue for judgments of learning. Psychonomic Bulletin & Review, 21, 445-453.

Sikström, S. & Jönsson, F. U. (2005). A model for stochastic drift in memory strength to account for judgments of learning. Psychological Review. 112(4), 932-950.

Son, L. K. & Metcalfe, J. (2005). Judgments of learning: Evidence for a two-stage model. Memory & Cognition, 33, 1116-1129.

Sundqvist, M.L., Todorov, I., Kubik, V., & Jönsson, F. U. (2012). Study for now, but judge for later: Delayed judgments of learning promote long-term retention. Scandinavian Journal of Psychology, 53, 450–454.

Underwood, G. (Coords.) (1966). Implicit cognition. Oxford, U.K.: Oxford University Press.

Vesonder, G. T. & Voss, J. F. (1985). On the ability to predict one’s own responses while learning. Journal of Memory & Language, 24, 363-376

Weaver, C.A. & Kelemen, W.L. (1997). Judgmens of learning at delays: Shifts in response patterns or increased metamemory accuracy? Psychological Science, 8, 318-321.

Weaver, C.A. & Kelemen, W.L. (2003). Processing similarity does not improve metamemory: Evidence against transfer-appropriate monitoring. Journal of Experimental Psychology: Learning, Memory, and Cognition, 29, 1058-1065

Weaver, C. A., Terrell, J. T., Krug, K. S., & Kelemen, W. L. (2008). The Delayed JOL Effect with very long delays: Evidence from flashbulb memories. In J. Dunlosky and R. A. Bjork (Eds.), A handbook of memory and metacognition (pp. 155-172). Hillsdale, NJ: Lawrence Erlbaum Associates.

How to Cite
Ruiz, M., & Arroyo, C. (2016). JOLer: A Java standalone application for simulating the Weaver & Kelemen’s judgment of learning (JOL) model. Anales De Psicología / Annals of Psychology, 32(3), 893-898.