Evidence that formative tests aid retention in medical education: an interview with Dr. Douglas Larsen

Dr Douglas LarsenPosted by John Kleeman

How do you slow the forgetting curve when teaching doctors, nurses and other medical specialists? I recently interviewed a prominent researcher in this field, Douglas Larsen MD M.Ed of Washington University in St Louis in the USA. Here is his explanation of how formative tests and quizzes help retain such learning.

What is your background and job role?

I am a practicing paediatric neurologist, and on the faculty at the Washington University School of Medicine in St Louis. I have a particular interest in medical education. In addition to my regular medical training, I have a Master’s degree in Medical Education through the University of Cincinnati. I am responsible for medical education in paediatric neurology at Washington University.

How did you get interested in assessments and testing?

During my time at Washington University, I became familiar with the work done by the psychologist, Professor Roediger (see my interview with Professor Roediger here). His work on what is known as “test-enhanced learning” – looking at how retrieval practice through quizzes and tests produces better retention of information than spending the same time studying — has practical applications in medical education. We worked together to do some applied studies. At the time, the longest retrieval intervals that had been studied had been a month to 6 weeks, but obviously for practical use in education, we’ve got to demonstrate that retrieval benefits last for much longer.

Our first study published in 2009 (see the blog article Evidence from Medical Education that Quizzes Do Slow the Forgetting Curve) investigated test-enhanced learning within medical resident education using a 6-month retrieval interval. We were able to demonstrate that the repeated taking of tests enhanced retention of information in educationally meaningfully intervals.

Can you explain the concept of how tests or quizzes enhance retention of learning in medical education?

The concept of test-enhanced learning is that repeated retrieval practice of information improves retention versus simply exposure to information. In very basic terms, you can think of it as active processing versus passive processing. If you compare reading a book to taking a test, then even though when reading a book, you may be thinking about the material, you are still essentially only required to recognise information. When you take a test, you have to generate and retrieve information from your memory.

Just like riding a bike, you can’t learn to ride a bike by watching someone: you have to actually do it. Similarly, in order for you to retrieve information, you need to practice doing it.

And how does this actually work inside the brain?

As far as the neural mechanism, we don’t have direct evidence at this time, but I would hypothesize that there are different circuits evolved for recognition versus retrieval. When you practice retrieving information, you are strengthening those circuits for retrieval, whereas you don’t necessarily have that from just exposure to information.

Does this just apply to facts (for instance about anatomy) or does it apply to concepts and diagnosis?

That’s a good question; ultimately our goal in medical education is not to get people to pass tests but to have them take care of patients. We’ve been looking at this in a recent study (just published online by Advances in Health Studies Education, see here or here) where we compared repeated studying, repeated written tests and repeated working with simulated patients. What we saw was that after a 6-month retention interval, there was much higher retention from the students who’d worked with simulated patients, but at the same time taking written tests gave better retention than just studying.

This indicates that test-enhanced learning does apply to broader activities. Recollection of factual information is important, but it’s important in an applied setting with patients. This tells us there needs to be close alignment between our objectives (what we want students to actually do) and how we test them.

When I talk to people about how quizzes and tests enhance retention, they sometimes say doctors and other medical workers need to do much more than just learn facts. What is your view on this?

When you look at the expertise literature in medicine, when researchers look at what differentiates expert clinicians from novice clinicians, they found that it was not so much a difference in problem-solving techniques, as it was in knowledge base and application of that knowledge. So learning knowledge and remembering knowledge is an important foundation of medical expertise. It is important to understand that we still have to learn facts and remember the facts.

But people are right, we need to be looking at the end product and the complex skills we want our students to learn. There are many skills where written testing cannot help, and simulation is more appropriate. Deliberate practice is critical for learning in simulation, and there are studies which show that using simulations for procedures like cardiac resuscitation and catheter insertion can improve retention. The way I like to think about this is that there are multiple different ways in which we can encourage retrieval – through written tests, through simulations or working with simulated or real patients. The key is to ensure retrieval practice and not just passive exposure.

Simulation or time with simulated patients is expensive and time-consuming. In future research, we will need to look at whether we can reduce the amount of simulation and increase the amount of written testing, and still get the same clinical benefits.


Next week, we’ll publish the second part of this interview where Dr. Larsen gives five principles for effective use of formative quizzes and tests within medical education.

Evidence from Medical Education that Quizzes Do Slow the Forgetting Curve

Posted by John Kleeman

I’ve recently come across a really exciting set of results from a study of doctors-in-training that show quizzes really do slow down the forgetting curve.

To remind you, once we learn something, we start to follow a downwards “forgetting curve”. Psychology research says that taking quizzes and tests can reduce the forgetting curve by giving us retrieval practice, which helps retain information. But it’s rare to see this shown in action quite so vividly as in an experiment conducted with doctors-in-training and published in the respected Medical Education journal. (The citation is: Larsen, D. P., Butler, A. C. and Roediger III, H. L. (2009), Repeated testing improves long-term retention relative to repeated study: a randomised controlled trial. Medical Education, 43: 1174–1181.)

In this experiment, Dr Douglas Larsen and colleagues Butler and Roediger from Washington University in St Louis divided some doctors-in- training into two groups of about 20 people each. The doctors were learning about two medical topics : status epilepticus  and myasthenia gravis, abbreviated to SE and MG below.

Group A did the following:

  • Study session on both SE and MG
  • Quiz on SE immediately after the session and equal re-study time spent on MG
  • 2 weeks after study session, quiz on SE and equal re-study time on MG
  • 4 weeks after study session, quiz on SE and equal re-study time on MG
  • 6 months later, test on both SE and MG

Group B did the same with SE and MG switched round, ie

  • Study session on both SE and MG
  • Quiz on MG immediately after the session and equal re-study time spent on SE
  • 2 weeks after study session, quiz on MG and equal re-study time on SE
  • 4 weeks after study session, quiz on MG and equal re-study time on SE
  • 6 months later, test on both SE and MG

If spending time on re-study and taking a quiz have equal benefit to retention, then you would expect that both groups would perform about the same on the final test.

But if taking a quiz does actually aid retention and slow down forgetting, then you would expect that Group A would do better on SE than group B, and that group B would do better on MG than group A.

So what did happen? Here are the results for topic SE; you can see that group A initially scored an average of 78% and then knowledge reduced over time to 42% on the final test. However group B got a final score of 31%, much lower.

Results for SE

And here are the results for topic MG; group B, who did the earlier quizzes/tests on MG, scored an average of 36% on the final test, but group A, who’d just done the studying, got 19%.

Results for MG

These results show very clearly that taking the quizzes/tests helped people retain information vs. spending the same amount of time re-studying. And the study gives a dramatic picture of what happens in real learning – how people do forget, but also how quizzes and tests can reduce this.