Professor Roddy Roediger on applying the retrieval practice effect to creating and administering assessments

Posted by John Kleeman

In the first part of this interview, I asked Professor Roediger to explain how quizzes and tests give retrieval practice that helps you learn.  Now, he moves on to giving some practical advice on how to use this effectively when creating and administering assessments.

Roddy Roediger

Is it better to have formal quizzes or self-retrieval practice?

They both work great, but there is a puzzle inherent in using self-testing types of retrieval practice.

In self-testing, students can base their studying on their self-knowledge. That is, we think we can accurately judge what we know and don’t know, so we will practice the material we don’t know until we know it. However, one outcome that cognitive psychologists have repeatedly shown is that often our own judgments of learning are not a sure guide to what we really know (as measured on a test).

Jeff Karpicke at Purdue University has done experiments comparing cases where learning is put under students’ control compared to that of a computer algorithm that requires the same schedule of testing for all students. You might imagine that if you let people test themselves by giving them a stack of cards and saying, “Just learn this material” until you are sure you know it, that they could outpace a computer program. After all, the student has awareness of his or her knowledge states to guide study whereas the computer does not. The computer might just have each person practice the fact three times. Still, what Karpicke finds is that when people are tested some days later, the computer schedule leads to better retention than when people are permitted to study according to their own schedule. The computer schedule that tests everyone on everything three times leads to better long-term learning.

Karpicke finds that when you put the ability to restudy and test under the students’ control, they usually do not test themselves enough. They retrieve something once and think they have it permanently without realizing that repeated testing is the key to long-term memory. Getting it once isn’t enough. You need repeated retrieval practice for something critical. If you retrieve it many times, the fact becomes much more easily retrieved in the future. So people who use retrieval practice often don’t do it that effectively, because they stop before they should.

How many times should one get people to retrieve things, and how soon after learning?

We’re just exploring this area – it probably depends on what kinds of material you’re trying to learn. Mary Pyc and Katherine Rawson at Kent State University showed that for simple things like foreign language vocabulary, retrieving about 5 to 7 times is about right — if you test people a week later you wouldn’t see much difference between having tested people 7 times or 10 times, but you do see gains going up to the range of 5-7 times. After that it just levels off. But most people would only practice once or twice, so the idea of going up to 5 or 7 retrievals seems like too much to many people. Of course, to keep knowledge at your mental fingertips, you would need continued spaced retrieval practice, too.

So would you advise people training sales people or factory workers to give people retrieval practice after training?

Yes, absolutely. Giving quizzes, tests or other retrieval practice will help people retain learning. Again, spaced retrieval is the key.

Besides too little retrieval practice, another mistake trainers sometimes make (in sports, in education, in the military, in industry) is to train people using blocks of practice on the same task. If you have 5 tasks you have to train a person to do, often teachers will train people up on task one and get them to practice until they get to 100% performance. They then train task two and bring them to 100%, and so forth. That’s fine — you get students up to speed quickly with this kind of training — but when you test them after a long delay, they also show rapid forgetting. The better training technique for long-term retention is to interleave the practice; that is, practice task one for a little while, then go to task three, then go to two and so forth.  That slows down initial learning and frustrates learners a bit; it frustrates teachers too because they don’t see a great initial improvement. However, many studies show that when you provide this interleaved practice, where you have to switch back and forth, it leads to much better retention later on. When you are practicing retrieval, or even if you’re practicing a motor skill, it’s better to skip around between tasks than to train on tasks one by one.

Is that an example of desirable difficulty?

Yes! (A desirable difficulty is a difficulty introduced during learning that slows initial learning but increases long-term retention and transfer.)

That’s really interesting….How important and useful is feedback?

Feedback is critical in the testing world. Taking a test when you can’t remember something is of no use, but if you get feedback, that can help a lot. For anything difficult, especially for tests after a delay, you need feedback. Feedback never hurts!

If you’re at a cocktail party and you can’t remember someone’s name after being introduced it’s hard to get feedback. But in most cases — if you’re a sales person learning your products or a student learning about the muscular system — you can arrange to get feedback, and I always advise people to get feedback. We’ve never seen studies where it hurts; it only helps.

In some cases, such as with multiple choice and true/false tests, we have seen some negative consequences of delivering tests without feedback. Suppose you answer a question with choice D, and you’re sure that is correct. Well, suppose it is actually wrong. We know people learn from answering questions on tests, so if you’ve responded and made an error, you will have stamped that error into memory. Because retrieval enhances learning, you will continue remembering that error. So especially in tests like true/false and multiple choice, where error is invited, feedback is very important. In fact, studies by Andrew Butler of Duke University show that potential negative effects of multiple-choice tests can be completely removed by providing feedback soon after the tests. As long as you give feedback on multiple choice quizzes and tests, they provide learning benefits.

What are you looking into now and what research still needs to be done?

There’s a lot to be done. Although the first experiments showing the retrieval practice or testing effect go back 100 years ago, it’s only been in the last 10 years that researchers have really begun to dig into the issues and have started exploring the parameters. Many researchers besides me are doing work on retrieval practice. One question being asked in my own lab now, by Adam Putnam, is, “Does it matter how you do the retrieval?” Does it matter if you write down your answer, type it into a computer, say it out loud or just think it? Is thinking the answer as effective as writing or typing it for later retention?  Right now, the early evidence is that they’re all about the same…. but we’re just beginning this line of work.

You can see more on Professor Roediger’s work at http://psych.wustl.edu/memory/publications/.  Research by Professor Roediger and colleagues is sponsored by the James S. McDonnell Foundation – a foundation left by James McDonnell, the famous aircraft pioneer, who had wide ranging interests including psychology. He is also sponsored by the Cognition and Student Learning Program of the U.S. Department of Education and by Dart Neuroscience.

5 Responses to “Professor Roddy Roediger on applying the retrieval practice effect to creating and administering assessments”

  1. […] week we’ll publish the second part of the interview, in which Professor Roediger gives practical advice for people seeking to use the […]

  2. Silvester Draaijer says:

    John,

    I really love your post about the work of Roediger III. I already encountered his work some years ago, then he showed up in the NYT and then the ball got really rolling. Your interview is added value to this stream of attention.

    I would like to add something relating to feedback. Maybe you could talk in a post about feedback, based on review work of Hattie and Timperley: Hattie, J., & Timperley, H. (2007). The Power of Feedback. Review of Educational Research, 77(1), 81-112. doi:10.3102/003465430298487. I tried to put their article into a Concept Map that you can find here: http://www.mywebspiration.com/view/362724ac2c5.

    Kind regards,

    Silvester

  3. […] Professor Roddy Roediger on applying the retrieval practice effect to creating and administering ass… […]

  4. […] Professor Roddy Roediger on applying the retrieval practice effect to creating and administering ass… […]

  5. […] at Washington University in St Louis, Missouri and several of his colleagues across the U.S. (Click here to read one of John’s interviews with Professor […]

Leave a Reply