Evidence that formative tests aid retention in medical education: an interview with Dr. Douglas Larsen

Dr Douglas LarsenPosted by John Kleeman

How do you slow the forgetting curve when teaching doctors, nurses and other medical specialists? I recently interviewed a prominent researcher in this field, Douglas Larsen MD M.Ed of Washington University in St Louis in the USA. Here is his explanation of how formative tests and quizzes help retain such learning.

What is your background and job role?

I am a practicing paediatric neurologist, and on the faculty at the Washington University School of Medicine in St Louis. I have a particular interest in medical education. In addition to my regular medical training, I have a Master’s degree in Medical Education through the University of Cincinnati. I am responsible for medical education in paediatric neurology at Washington University.

How did you get interested in assessments and testing?

During my time at Washington University, I became familiar with the work done by the psychologist, Professor Roediger (see my interview with Professor Roediger here). His work on what is known as “test-enhanced learning” – looking at how retrieval practice through quizzes and tests produces better retention of information than spending the same time studying — has practical applications in medical education. We worked together to do some applied studies. At the time, the longest retrieval intervals that had been studied had been a month to 6 weeks, but obviously for practical use in education, we’ve got to demonstrate that retrieval benefits last for much longer.

Our first study published in 2009 (see the blog article Evidence from Medical Education that Quizzes Do Slow the Forgetting Curve) investigated test-enhanced learning within medical resident education using a 6-month retrieval interval. We were able to demonstrate that the repeated taking of tests enhanced retention of information in educationally meaningfully intervals.

Can you explain the concept of how tests or quizzes enhance retention of learning in medical education?

The concept of test-enhanced learning is that repeated retrieval practice of information improves retention versus simply exposure to information. In very basic terms, you can think of it as active processing versus passive processing. If you compare reading a book to taking a test, then even though when reading a book, you may be thinking about the material, you are still essentially only required to recognise information. When you take a test, you have to generate and retrieve information from your memory.

Just like riding a bike, you can’t learn to ride a bike by watching someone: you have to actually do it. Similarly, in order for you to retrieve information, you need to practice doing it.

And how does this actually work inside the brain?

As far as the neural mechanism, we don’t have direct evidence at this time, but I would hypothesize that there are different circuits evolved for recognition versus retrieval. When you practice retrieving information, you are strengthening those circuits for retrieval, whereas you don’t necessarily have that from just exposure to information.

Does this just apply to facts (for instance about anatomy) or does it apply to concepts and diagnosis?

That’s a good question; ultimately our goal in medical education is not to get people to pass tests but to have them take care of patients. We’ve been looking at this in a recent study (just published online by Advances in Health Studies Education, see here or here) where we compared repeated studying, repeated written tests and repeated working with simulated patients. What we saw was that after a 6-month retention interval, there was much higher retention from the students who’d worked with simulated patients, but at the same time taking written tests gave better retention than just studying.

This indicates that test-enhanced learning does apply to broader activities. Recollection of factual information is important, but it’s important in an applied setting with patients. This tells us there needs to be close alignment between our objectives (what we want students to actually do) and how we test them.

When I talk to people about how quizzes and tests enhance retention, they sometimes say doctors and other medical workers need to do much more than just learn facts. What is your view on this?

When you look at the expertise literature in medicine, when researchers look at what differentiates expert clinicians from novice clinicians, they found that it was not so much a difference in problem-solving techniques, as it was in knowledge base and application of that knowledge. So learning knowledge and remembering knowledge is an important foundation of medical expertise. It is important to understand that we still have to learn facts and remember the facts.

But people are right, we need to be looking at the end product and the complex skills we want our students to learn. There are many skills where written testing cannot help, and simulation is more appropriate. Deliberate practice is critical for learning in simulation, and there are studies which show that using simulations for procedures like cardiac resuscitation and catheter insertion can improve retention. The way I like to think about this is that there are multiple different ways in which we can encourage retrieval – through written tests, through simulations or working with simulated or real patients. The key is to ensure retrieval practice and not just passive exposure.

Simulation or time with simulated patients is expensive and time-consuming. In future research, we will need to look at whether we can reduce the amount of simulation and increase the amount of written testing, and still get the same clinical benefits.


Next week, we’ll publish the second part of this interview where Dr. Larsen gives five principles for effective use of formative quizzes and tests within medical education.

4 Responses to Evidence that formative tests aid retention in medical education: an interview with Dr. Douglas Larsen

  1. Thomas Garrod says:

    The good doctor is addressing basic cognitive theory principles, such as the dual encoding resulting from elaborative rehearsal. He touts tests for this purpose, but this is decidedly old school. The tests are generally designed to measure total recall. They allow learners to test their retention, and this will support recall of facts later.

    A better strategy to retention involves a more social constructivist model. To find better solutions, we need to step out of the old-school teaching theories. The doctor appears more comfortable with the passive lecture model and uses tests to support retention. Better learning approaches get learners talking to each other. They create a community of practice that supports the course and keeps the discussion open after the class.

    If you want to advance your learning strategies, talk to an instructional designer not a teacher. Teachers are experts in learning delivery and classroom management. They are not generally trained in the underlying theories supporting learning strategy design. While teachers learn about behavior and teaching pedagogy, ISDs learn about cognitive theory (how we process information), presentation theory and metacognitive support (how user interface and information structuring supports or undermines information processing), and general principles of learning strategy such as leveraging prior learning, scaffolding, and providing how-it-works information.

    Thomas Garrod, M.Ed. (the degree looks the same, but it isn’t about teaching)

  2. […] the first part of this interview, Dr. Douglas Larsen, an expert in medical education at the Washington University […]

  3. Dr. Douglas Larsen says:

    Mr. Garrod brings up some important points. Social constructivism and situated cognition have made important contributions to our understanding of educational systems and learning. However, to restrict oneself to a certain set of tools and not use all of the evidence-based interventions that we have at our disposal also leads to sub-optimal results.

    Retrieval practice is one of the most robust, replicable, and generalizable educational interventions in the cognitive psychology literature. Because this literature has literally exploded over the last six or seven years it is difficult to call it “old-school.” Retrieval practice does not simply endorse the rote memorization of facts. Studies have shown that retrieval practice through tests helps students to form their own organizational models (see Zaromb and Roediger 2010), promotes transfer of knowledge to new situations (see Butler 2010 and Larsen et al. 2012), encourages concept learning (Jacoby et al. 2010), as well as increases metacognitive awareness of learning (see Jacoby et al. 2010 and Agrawal et al. 2012). Educators would be foolhardy to reject the wealth of evidence supporting retrieval practice.

    As I explained in my interview with Mr. Kleeman, retrieval practice does not have to be limited to written tests. Retrieval practice takes many forms. The key is for educators to actively plan for it. Indeed, the principles of instructional design endorse the importance of providing adequate practice for whatever type of learning is the object of the educational situation (see Smith and Ragan 2005). I reject Mr. Garrod’s assertion that I endorse passive lectures with the use of tests to make up for their deficiencies. Active learning strategies and retrieval practice are complementary, not exclusionary. Great educators (whether they be classroom teachers or instructional designers) choose the best evidence-based tools for their objectives no matter what label is applied.

    Agrawal, S., Norman, G. R., and Eva, K. W. (2012). Influences on medical students’ self-regulated learning after test completion. Medical Education, 46, 326-335.

    Butler, A. C. (2010). Repeated testing produces superior transfer of learning relative to repeated studying. Journal of Experimental Psychology: Learning, Memory, and Cognition, 36, 1118-1133.

    Jacoby, L. L., Wahlheim, C. N., and Coane, J. H. (2010). Test-enhanced learning of natural concepts: effects on recognition memory, classification, and metacognition. Journal of Experimental Psychology: Learning, Memory, and Cognition, 36, 1441-1451.

    Larsen, D. P., Butler, A. C., Lawson, A. L., and Roediger, H. L., III (2012). The importance of seeing the patient: Test-enhanced learning with standardized patients and written tests improves clinical application of knowledge. Advances in Health Sciences Education, doi: 10.1007/s10459-012-9379-7 (published online ahead of print).
    Smith, P. L., and Ragan, T. J. (2005). Instructional Design. Hoboken, NJ: John Wiley and Sons, Inc.

    Zaromb, F.M. and Roediger, H.L. (2010). The testing effect in free recall is associated with enhanced organization processes. Memory and Cognition, 38, 995-1008.

  4. Might I add:

    Kerfoot, B, P (2006) SPACED EDUCATION. Interactive Spaced-Education to Teach the Physical Examination: A randomized Controlled Trial.

Leave a Reply

Your email address will not be published. Required fields are marked *