Evidence that formative tests aid retention in medical education: an interview with Dr. Douglas Larsen

Dr Douglas LarsenPosted by John Kleeman

How do you slow the forgetting curve when teaching doctors, nurses and other medical specialists? I recently interviewed a prominent researcher in this field, Douglas Larsen MD M.Ed of Washington University in St Louis in the USA. Here is his explanation of how formative tests and quizzes help retain such learning.

What is your background and job role?

I am a practicing paediatric neurologist, and on the faculty at the Washington University School of Medicine in St Louis. I have a particular interest in medical education. In addition to my regular medical training, I have a Master’s degree in Medical Education through the University of Cincinnati. I am responsible for medical education in paediatric neurology at Washington University.

How did you get interested in assessments and testing?

During my time at Washington University, I became familiar with the work done by the psychologist, Professor Roediger (see my interview with Professor Roediger here). His work on what is known as “test-enhanced learning” – looking at how retrieval practice through quizzes and tests produces better retention of information than spending the same time studying — has practical applications in medical education. We worked together to do some applied studies. At the time, the longest retrieval intervals that had been studied had been a month to 6 weeks, but obviously for practical use in education, we’ve got to demonstrate that retrieval benefits last for much longer.

Our first study published in 2009 (see the blog article Evidence from Medical Education that Quizzes Do Slow the Forgetting Curve) investigated test-enhanced learning within medical resident education using a 6-month retrieval interval. We were able to demonstrate that the repeated taking of tests enhanced retention of information in educationally meaningfully intervals.

Can you explain the concept of how tests or quizzes enhance retention of learning in medical education?

The concept of test-enhanced learning is that repeated retrieval practice of information improves retention versus simply exposure to information. In very basic terms, you can think of it as active processing versus passive processing. If you compare reading a book to taking a test, then even though when reading a book, you may be thinking about the material, you are still essentially only required to recognise information. When you take a test, you have to generate and retrieve information from your memory.

Just like riding a bike, you can’t learn to ride a bike by watching someone: you have to actually do it. Similarly, in order for you to retrieve information, you need to practice doing it.

And how does this actually work inside the brain?

As far as the neural mechanism, we don’t have direct evidence at this time, but I would hypothesize that there are different circuits evolved for recognition versus retrieval. When you practice retrieving information, you are strengthening those circuits for retrieval, whereas you don’t necessarily have that from just exposure to information.

Does this just apply to facts (for instance about anatomy) or does it apply to concepts and diagnosis?

That’s a good question; ultimately our goal in medical education is not to get people to pass tests but to have them take care of patients. We’ve been looking at this in a recent study (just published online by Advances in Health Studies Education, see here or here) where we compared repeated studying, repeated written tests and repeated working with simulated patients. What we saw was that after a 6-month retention interval, there was much higher retention from the students who’d worked with simulated patients, but at the same time taking written tests gave better retention than just studying.

This indicates that test-enhanced learning does apply to broader activities. Recollection of factual information is important, but it’s important in an applied setting with patients. This tells us there needs to be close alignment between our objectives (what we want students to actually do) and how we test them.

When I talk to people about how quizzes and tests enhance retention, they sometimes say doctors and other medical workers need to do much more than just learn facts. What is your view on this?

When you look at the expertise literature in medicine, when researchers look at what differentiates expert clinicians from novice clinicians, they found that it was not so much a difference in problem-solving techniques, as it was in knowledge base and application of that knowledge. So learning knowledge and remembering knowledge is an important foundation of medical expertise. It is important to understand that we still have to learn facts and remember the facts.

But people are right, we need to be looking at the end product and the complex skills we want our students to learn. There are many skills where written testing cannot help, and simulation is more appropriate. Deliberate practice is critical for learning in simulation, and there are studies which show that using simulations for procedures like cardiac resuscitation and catheter insertion can improve retention. The way I like to think about this is that there are multiple different ways in which we can encourage retrieval – through written tests, through simulations or working with simulated or real patients. The key is to ensure retrieval practice and not just passive exposure.

Simulation or time with simulated patients is expensive and time-consuming. In future research, we will need to look at whether we can reduce the amount of simulation and increase the amount of written testing, and still get the same clinical benefits.


Next week, we’ll publish the second part of this interview where Dr. Larsen gives five principles for effective use of formative quizzes and tests within medical education.

6 + 6 is more than 12 : How Online Assessments enhance Distance Learning

Posted by John Kleeman

What is stopping distance learning from being more prevalent, in higher education and elsewhere?

I recently attended the Guide Conference in Rome, Italy, and was impressed by the many presentations on distance learning and e-learning from participants from 46 different countries. This provided a genuine world view on these topics!. At the conference, I gave a presentation (co-authored with Eric Shepherd) about how technology enhancements in online assessment might be the key factor to allow distance learning to become mainstream. I’d like to share these ideas here.

Below is a picture of the typical technology adoption lifecycle. Initially new technology is used by Innovators who use things because they are new. Then visionaries or Early Adopters see the potential business value of new technology and are willing to use it in spite of issues and problems. But often there is a Chasm before the majority of users adopt a technology; the majority will only use technology if their peers are already using it and if it meets all their needs.

I’d suggest that distance learning in higher education might be in the Chasm today – it’s being used by visionaries but not yet by the majority.

Technology Adoption Lifecycle

In the presentation, I pick out six key assessments in distance learning:

  • Pretests
  • Formative quizzes
  • Practice tests
  • Course evaluations
  • Quizzes to slow the forgetting curve
  • Exams

And six key, recent technological advances:

  • Embedding assessments on the same page as learning
  • Mobile delivery of assessments
  • Remote video monitoring
  • Easy to use learning platforms
  • Easy to translate assessments
  • The Cloud easing deployment

I wonder if the combination of these assessments and technology advances could make a significant difference for distance learning success. In particular, the ability to do higher stakes exams more safely at a distance makes distance learning more credible. And the increased knowledge, evidence and engagement given by frequent online assessments greatly aids learning by reducing the “transactional distance” between instructor and learner.

You can see the presentation embedded below or else view it by clicking on the title:

6+6 is more than 12

View more presentations from Questionmark

I’d love to receive your comments. Is distance learning already mainstream, and if not, what will it take for the majority to adopt it?