Evidence that formative tests aid retention in medical education: an interview with Dr. Douglas Larsen

Dr Douglas LarsenPosted by John Kleeman

How do you slow the forgetting curve when teaching doctors, nurses and other medical specialists? I recently interviewed a prominent researcher in this field, Douglas Larsen MD M.Ed of Washington University in St Louis in the USA. Here is his explanation of how formative tests and quizzes help retain such learning.

What is your background and job role?

I am a practicing paediatric neurologist, and on the faculty at the Washington University School of Medicine in St Louis. I have a particular interest in medical education. In addition to my regular medical training, I have a Master’s degree in Medical Education through the University of Cincinnati. I am responsible for medical education in paediatric neurology at Washington University.

How did you get interested in assessments and testing?

During my time at Washington University, I became familiar with the work done by the psychologist, Professor Roediger (see my interview with Professor Roediger here). His work on what is known as “test-enhanced learning” – looking at how retrieval practice through quizzes and tests produces better retention of information than spending the same time studying — has practical applications in medical education. We worked together to do some applied studies. At the time, the longest retrieval intervals that had been studied had been a month to 6 weeks, but obviously for practical use in education, we’ve got to demonstrate that retrieval benefits last for much longer.

Our first study published in 2009 (see the blog article Evidence from Medical Education that Quizzes Do Slow the Forgetting Curve) investigated test-enhanced learning within medical resident education using a 6-month retrieval interval. We were able to demonstrate that the repeated taking of tests enhanced retention of information in educationally meaningfully intervals.

Can you explain the concept of how tests or quizzes enhance retention of learning in medical education?

The concept of test-enhanced learning is that repeated retrieval practice of information improves retention versus simply exposure to information. In very basic terms, you can think of it as active processing versus passive processing. If you compare reading a book to taking a test, then even though when reading a book, you may be thinking about the material, you are still essentially only required to recognise information. When you take a test, you have to generate and retrieve information from your memory.

Just like riding a bike, you can’t learn to ride a bike by watching someone: you have to actually do it. Similarly, in order for you to retrieve information, you need to practice doing it.

And how does this actually work inside the brain?

As far as the neural mechanism, we don’t have direct evidence at this time, but I would hypothesize that there are different circuits evolved for recognition versus retrieval. When you practice retrieving information, you are strengthening those circuits for retrieval, whereas you don’t necessarily have that from just exposure to information.

Does this just apply to facts (for instance about anatomy) or does it apply to concepts and diagnosis?

That’s a good question; ultimately our goal in medical education is not to get people to pass tests but to have them take care of patients. We’ve been looking at this in a recent study (just published online by Advances in Health Studies Education, see here or here) where we compared repeated studying, repeated written tests and repeated working with simulated patients. What we saw was that after a 6-month retention interval, there was much higher retention from the students who’d worked with simulated patients, but at the same time taking written tests gave better retention than just studying.

This indicates that test-enhanced learning does apply to broader activities. Recollection of factual information is important, but it’s important in an applied setting with patients. This tells us there needs to be close alignment between our objectives (what we want students to actually do) and how we test them.

When I talk to people about how quizzes and tests enhance retention, they sometimes say doctors and other medical workers need to do much more than just learn facts. What is your view on this?

When you look at the expertise literature in medicine, when researchers look at what differentiates expert clinicians from novice clinicians, they found that it was not so much a difference in problem-solving techniques, as it was in knowledge base and application of that knowledge. So learning knowledge and remembering knowledge is an important foundation of medical expertise. It is important to understand that we still have to learn facts and remember the facts.

But people are right, we need to be looking at the end product and the complex skills we want our students to learn. There are many skills where written testing cannot help, and simulation is more appropriate. Deliberate practice is critical for learning in simulation, and there are studies which show that using simulations for procedures like cardiac resuscitation and catheter insertion can improve retention. The way I like to think about this is that there are multiple different ways in which we can encourage retrieval – through written tests, through simulations or working with simulated or real patients. The key is to ensure retrieval practice and not just passive exposure.

Simulation or time with simulated patients is expensive and time-consuming. In future research, we will need to look at whether we can reduce the amount of simulation and increase the amount of written testing, and still get the same clinical benefits.

 

Next week, we’ll publish the second part of this interview where Dr. Larsen gives five principles for effective use of formative quizzes and tests within medical education.

Feedback in Questionmark Live

Posted by Jim Farrell

As I started thinking about what I wanted to blog about, I couldn’t get past the podcast done by our very own Joan Phaup and Dr. Will Thalheimer of Work-Learning Research on the use of feedback. One of the most powerful features in Questionmark Live is the ability to leave choice-based feedback. I will likely have many blog posts on this topic and Dr. Thalheimer’s white paper, but let’s start at the beginning:

Retrieval is more important than feedback. The role that feedback plays is to support retrieval.

This statement by Will seems simple, but it helps to understand how to write good feedback. There are so many things to think about when creating feedback in a question.

  • When is the retrieval opportunity presented?
  • What is the feedback for a correct answer?
  • What is the feedback for an incorrect answer?

How does Questionmark Live fit into this? Well, it is pretty easy to write feedback for late-in-learning retrieval since you are only trying to get the learner back on track. It is the early-in-learning feedback that needs to be more extensive so it can help the learner develop pathways to information to support later retrieval. Allowing a subject matter expert (SME) to create extensive feedback in Questionmark Live will ensure that your feedback is detailed and accurate. No one is expecting the SME to be an expert in question writing. You may need to tweak the question once you bring it into Perception, but your feedback will be far more powerful if you glean it from someone who knows about a subject in depth.

 

Multiple Choice Question with Feedback Showing

 

I really encourage you to read Dr. Thalheimer’s white paper to help you use feedback to improve the learning process.