Effective assessment feedback needs motive, opportunity and means

John Kleeman HeadshotPosted by John Kleeman

Assessment feedback, whether it relates to a question, a topic or an entire assessment, has tremendous value – but only if learners make practical use of it! I’d like to share some solid approaches about feedback and talk about how a couple of Questionmark reports can help you put them into practice.

From Andrew Morgan (quoted in Valerie Shute’s excellent ETS research report on feedback), we get the concept that to be effective and useful, feedback needs the following:

  • Motive – the learner wants to follow the feedback
  • Opportunity – the learner has it in time to use it: it’s not given too late for action
  • Means – the learner is able to use it: the feedback is actionable

Another good way to think about feedback comes from Dr Steve Draper of the University of Glasgow School of Psychology in his presentation at eAssessment Scotland in 2012:

“There is no point in giving feedback to a learner unless the learner acts on it: does something concrete and differently because of it”.

Feedback needs to be processed to be usefulFeedback that the learner doesn’t read isn’t valuable.

Feedback that the learner reads but doesn’t process isn’t valuable.

You must get the learner to evaluate the feedback and adjust their thinking as a result of the feedback, i.e. process the feedback and do something with it.

I’ve been wondering about how you can apply these concepts using the Questionmark coaching report when presenting an assessment score as feedback.

Most learners are motivated to use their score achieved in a test as feedback; they want to get a high scores next time. And if they can take a test again, they have the opportunity to use the feedback. But a score on its own is just a number. How can you help learners use their scores as catalysts for action?

Clearly, a score is more valuable if it can be compared to something, and there are three obvious comparisons:

  • Ipsative, comparing score to previous attempts: have you done better than last time?
  • Criterion referenced, comparing score to a benchmark: have you reached the desired standard?
  • Normative, comparing score to how others do: how do you compare to your peers?

Questionmark’s Transcript report lets learners view all their results and see how they improve or change over time. And Questionmark’s Coaching report includes the concept of benchmarks – you can set a benchmark for the assessment and for each topic. What you may not know is that the Coaching report allows you to compare a score against the average of a group of other test-takers. You can define the group of people by a variety of demographics and then display how the participant compares against their scores. This screenshot shows how to set this up:

Setting a comparison in the coaching report

Giving learners information about how they compare to others can be a powerful motivator; I encourage you to explore this capability of the Questionmark Coaching report.

For more on general concepts of processing feedback, see Steve Draper’s interesting page here. Questionmark users can see more information on the Coaching report comparison options on the Questionmark support site. And if you want to hear more from me about assessment feedback, I’ll be speaking about it at the Questionmark user conference in March.

The impact of feedback on learning and retention

Joan Phaup HeadshotPosted by Joan Phaup

Each year at the Questionmark Users Conference we like to include at least one breakout session relating to cognitive learning research – and 2013 is no exception.

John Kleeman, Questionmark’s founder and chairman, takes a special interest in learning research and has been focusing lately on the role feedback plays in improving the value of quizzes and tests.

John will lead a best practices session when we meet in Baltimore March 3 – 6, on Assessment Feedback – What Can We Learn from Psychology Research?

I spent a few minutes asking John about his presentation.

John Kleeman portrait

John Kleeman

What research have you been following on the effects of feedback on learning and retention?

My main role is as chairman of Questionmark, but I keep an active eye on relevant research, and I follow a number of researchers who are looking into how learning and retention work – and I’m particularly interested in how assessments fit into that. For example, I’ve been following Professor Roddy Roediger at Washington University in St Louis, Missouri and several of his colleagues across the U.S.

(Click here to read one of John’s interviews with Professor Roediger.)

What would you say are the key findings from this research?

What we kind of know but don’t always put into practice is that we forget a surprising amount of what we learn. People know about the forgetting curve as an idea, but don’t always think it applies to them! We think we are going to be better at remembering things later than we actually are. A quiz or test can force you into practicing retrieving and that makes it more likely for things to stick in your mind.

When you learn something – whether in a formal or informal context– you won’t remember a lot of it in a month or six months. Taking a quiz or test helps you retain that learning by providing retrieval practice and slowing the forgetting curve. If people take quizzes or tests, it slows down the forgetting curve – and quizzes with feedback slow down the forgetting curve more effectively than quizzes without feedback.

Will you discuss topic feedback and well as question feedback?

A lot of the research covers question feedback because it’s very easy to measure how well people do on a specific fact. But there is also evidence about topic feedback, and yes; I will be covering topic feedback as well as question feedback.

What would you like your audience to take away from your session?

I aim to practice what I preach, so I will use interactive techniques to help people remember what I talk about! I don’t want just to provide theory: I also want to give actionable ideas that people can apply to their Questionmark assessments to improve retention.

I’d like to add that I’ve found from talking with customers that the conference is a fantastic place to learn. People who come to the conference get a lot of formal learning – for instance by presentations from assessment experts and Questionmark staff who explain effective ways to use our technologies – but they also get a lot of informal learning from interacting with other users. I’m especially looking forward to presentations from our expert customers. Some of our case study presenters have been using our software for many years and have a lot of experience and wisdom to share. So I think I’ll learn a lot from those presentations myself!

You can save $100 if you register for the conference by January 18th. Check out the conference agenda and sign up soon!


Don’t believe quizzes and tests help retention? Heed the evidence

Posted by John Kleeman

I’ve been sharing the new evidence from cognitive psychology at several conferences in the last year. This peer-reviewed, scientific evidence shows that taking a quiz or test enhances learning and slows the forgetting curve. Although many people are enthused and fascinated, some are doubtful. And a few turn away because they regard cognitive psychology as being out of fashion.

Of course, it takes time for all new evidence to get accepted, and I’d like to share some of the questions comments and objections I’ve been hearing lately.

In case you’re not familiar with the evidence, here are the slides from my presentation at Online Educa in Berlin in December (or you can also see them on SlideShare), or check out related blog articles: here, here and here).


View more presentations from Questionmark

You could also come to my session at the Questionmark Users Conference in New Orleans in March 2012, where I’ll be talking about the benefits of spacing out learning and how assessments can help here.

So what are the objections I’ve been hearing recently? And how do I respond?

“We test our students too much already, we don’t need more tests”

Some people confuse the benefits of quizzes and tests as retrieval practice with the over-use of standardized tests in schools. They are two different things – and it can be helpful to use a word other than “tests” for retrieval practice, for instance quizzes or exercises.

“We don’t have time for more tests and quizzes”

A complaint can be that the course is too full to find space for more quizzes. But since the evidence shows that spending time taking a quiz helps learning more than re-studying a subject, it may be worth adjusting things. If you want your learners to retain what you teach, not just learn and forget it, you need to give quizzes or other retrieval practice

“We teach creativity or other higher level skills. not facts”

There are two answers to this. One is that the research implies that quizzes and tests encourage retention of concepts as well as facts. The other is that all disciplines need knowledge of basic facts and vocabulary to build creativity on top.

“How valid is the research?”

Some people would like to see more studies with larger numbers of participants and in the real world. Or they wonder whether increased forgetting of the material not tested counter-balances the positive effects of quizzes as retrieval practice. These are good questions; this research has mostly happened in the last few years, and I would like to see more of it. But the basic concepts go back hundreds of years. The evidence for quizzes directly helping learning, particularly if they include feedback, is strong.

“Our learners don’t like being tested”

Unfortunately the retrieval practice benefit of quizzes helps more in the long term than the short term. So there is a paradox that if you take a quiz or test you might feel it’s not helping when it is. This can be a challenge. Part of the answer is to make quizzes enjoyable and part is to educate learners on the benefits of retrieval practice.

“Our employees have privacy concerns about being tested”

There are ways to deal with this, for example by following a respected standard like ISO 10667. If privacy is a concern, you can make retrieval practice quizzes anonymous. This still gives learning benefit but makes it harder to track.

“Psychologists don’t know anything about pedagogy”

This may be true, but if psychologists can demonstrate (as they seem to in peer-reviewed studies) that learners retain information better in memory by learning in one way or another, then sensible instructors will learn from the research and apply it to their pedagogy.

“Cognitive psychology is not fashionable in education”

This is the one objection I don’t have a good answer to! I may be missing some of the history between the two disciplines, but it seems to me that we should base instruction and learning on what the evidence says and not on fashion or opinion. But if someone believes otherwise I’m not sure how to persuade them. Any suggestions?

Advice from Cognitive Psychologist Roddy Roediger on using retrieval practice to aid learning

Posted by John Kleeman

I am a keen admirer of the work of Professor Roddy Roediger, a cognitive psychologist who investigates how quizzes and tests directly aid learning by giving retrieval practice. I recently interviewed him and here is how he explains this effect and how we can apply this in practice.

Roddy Roediger

Could you explain a little about your background and how you moved into the memory field?

I have a Ph.D. in cognitive psychology from Yale University. I’ve always been interested in memory, and I was surprised to find there was an academic discipline devoted to studying remembering, so I naturally gravitated to that. I worked with Robert Crowder and Endel Tulving at Yale, two leading people in the field. Since then I have taught at Purdue University, the University of Toronto and Rice University. I am now James S. McDonnell Distinguished University Professor at Washington University in St. Louis.

Most of my career has been doing laboratory research trying to show factors that help or harm memory. In the 1990s I published a series of studies on illusions of memory – that is, on false memories  — how we can have very strong memories of something that either never happened or that happened quite differently from how we remember it. About 8 years ago I became interested in applying what we were learning about memory to education, and I started looking at factors that are important for learning and remembering but that are not well appreciated in education. One of these is retrieval practice, which is what happens when we test ourselves, or when we are given a test or quiz. When we actually retrieve information from memory, it’s a very potent enhancement to remembering it. We are much more likely to remember something again if we actively retrieve it than if we are passively exposed to it in restudying.

Is this the testing or quizzing effect  —  that if you learn something and answer questions on it, you are much more likely to retain it for the long term than if you don’t answer the questions?

Yes, absolutely. Making people actually think about material, to reconstruct it, to say it in their own words is much more effective than simply restudying it, yet many students don’t seem to appreciate this. If you ask students how they study to remember, their study strategy is typically re-reading and reviewing. That’s good up to a point, but it would be much better if they actively practiced retrieval, which is what a test requires them to do. If you haven’t constructed or answered practice questions, you won’t do as well on a test as students who have practiced.

A lot of our readers are in corporate training; does this apply in this field too? How should this affect people’s design of learning programs?

I think retrieval practice can have direct implications in the corporate world.

Let me give you an anecdote. One of the people I was talking with about this was skeptical. She was going to work on the train, reading the newspaper like she does every morning. She decided she’d put the paper down after each story and summarize it to herself mentally in her own words. When she got home that night, she asked her husband to test her on the stories she’d read.  And she did really well, surprising them both. Because after she’d read the stories, she’d retrieved them and put them in her own words in her mind.

So if you’re a sales person and you need to remember a lot of qualities of your product to go out and sell it, the best way to do it is to practice retrieving the information and consult your notes only when you fail to retrieve a critical piece of information. Then when you are with a customer you will know all the information. I talk to textbook sales people a lot. Some can walk in and tell me all about the books while others just get out their notes in their folders and show them to me. It’s so much more impressive when the the salespeople can look you in the eye and tell you about the book without having to refer to their notes.

How does this actually work inside the brain?

We don’t know the neural mechanisms yet, but i can tell you some factors that seem to be important.

There seems to be something about effortful retrieval that matters. If you have to put a bit of effort into the retrieval — if it’s harder to bring the fact out of memory — that helps. So imagine you are trying to remember a face or a name; say you meet someone and you want to remember her name. You might think it would be good to repeatedly retrieve the name immediately after you met her, but it is not. Repeated immediate retrieval is like rote rehearsal – and that doesn’t do very well. But if you space out your retrievals – so you do it right away after you meet the person (to make sure you have the name) and then you wait a while to try again and you keep trying at spaced intervals, you will remember the name much better. The delayed retrieval makes you expend a bit more effort. You want to make retrieval a little difficult for yourself, so something about retrieval effort does seem to matter.

Another way you can see that is if you have people read a passage and take a multiple-choice test. In a multiple-choice test you see all the alternatives and you see which one is familiar and correct. You will get a slight benefit in retention from that. But if you are given a short answer question and can actively retrieve the answer, you will get even more benefit, because you have to reproduce the information instead of just recognizing it.  Although both tests provide a benefit, research shows that more benefit accrues from a short answer test or quiz where you have to retrieve information than from a multiple choice or true/false one where you just have to recognize it.

Would that apply to other kinds of recall questions like putting a number as your answer or filling in a blank in a question?

Yes, it does. Fill-in-blank questions do provide the benefit. I assume the same would be true in remembering numbers, but I do not know any research on that topic yet.

What about with multiple response questions or matching questions?

We haven’t done the research in these areas, but we believe that questions that stimulate recall are superior to those that use recognition, but all retrieval practice is useful.

Does it just apply to learning knowledge and facts or does it apply to learning concepts and higher levels of learning?

It definitely applies to concepts. Let me give one example.

Larry Jacoby and his colleagues at Washington University study how people learn bird concepts like warbler or thrasher and so on. He had some people study examples of birds and which category they were in, and another group were given tests on birds and tried  to guess which category they belonged in (and then they got feedback).  So one group just studied birds with their category names whereas the other group learned them while being tested on the names. When he tested both groups a couple of days later, the people who’d had been tested while learning did better than those who’d just studied the examples and the categories. In the test, he showed novel examples that people hadn’t seen before, for instance a bird that belonged in the thrasher family but that had not been used in the practice phase, and the people who’d taken the tests did better. Answering the questions about the birds allowed them to grasp the concept better and generalize it to new examples.

By testing yourself, making mistakes and being corrected, you sharpen what you know about a concept.

So this sounds like a significant way that people can learn better that isn’t widely known. Why is that?

I don’t know! In his essay on memory, Aristotle said, “Exercise in repeatedly recalling a thing strengthens the memory.” Sir Francis Bacon and William James also knew the benefits of retrieval practice (or recitation, as it used to be called) and wrote about them. They didn’t have evidence, of course, except from their own experience. But the technique has mostly been lost from education and training.  In fact the idea of retrieval practice is pretty much derided in education because people in the U.S., at least, are so opposed to anything that smacks of testing.

Certainly testing can be misused; in the old school days there was an emphasis on rote memorization –students had to remember poetry, say, by heart. Educators later decried what they called this “kill and drill” approach to education and they got away from these techniques. That is good in part, because the philosophy behind rote memorization was misguided. Some educators a hundred years ago considered “Memory” to be a faculty of mind and to operate like a muscle. The idea was if a student practiced memorizing poetry, “the Memory” would become stronger and would be better at learning and remembering other things, like algebra. However, the mind simply does not work that way. Practicing one topic helps that topic but does not usually spill over to learning unrelated topics.

But on the other hand, with the de-emphasis on memorization, the benefits of active retrieval should not be lost, because active retrieval is a potent memory enhancer. When you see how children learn multiplication tables, they use flash cards with 6×4 on one side and 24 on the other, and teachers say, “Practice until you think you really know it. Practice until it’s completely automatic.” So teachers use retrieval practice for multiplication tables, but the idea that you can use it for much for complex ideas is not widely appreciated.

Testing has gotten a bad name in the educational community. Instead of thinking of testing as standardized testing to place people into groups, we need to see use of low-stakes quizzes in the classroom and self-testing outside the classroom as a study and learning strategy.

Next week we’ll publish the second part of the interview, in which Professor Roediger gives practical advice for people seeking to use the retrieval practice effect to help people learn.

Multiple choice quizzes help learning, especially with feedback

Posted by John Kleeman

I promised in an earlier blog entry to pass on my understanding of research in educational psychology about the unmediated or direct benefits of questioning, i.e., how answering questions helps people learn. I’ve recently read a 2008 paper by Butler and Roediger from the Memory Lab at Washington University in St. Louis, Missouri (see here for the 2008 paper and here for a 2010 review paper including the graph below), which includes some fascinating information on how multiple choice quizzes directly aid learning.

The researchers divided students randomly into four groups as follows

  • Study a subject, no quiz
  • Study a subject, take a multiple choice quiz, no feedback
  • Study a subject, take a multiple choice quiz, feedback after each question
  • Study a subject, take a multiple choice quiz, feedback at the end of the quiz

They then tested all the groups a week later and got the results below.

Chart showing quizzes give better retention

As you can see, the students who had taken a quiz (or test as the authors describe it) got better results on average than those who hadn’t taken a quiz. This is expected due to the general principle that answering questions gives retrieval practice, which helps you to be able to recall things later and so helps learning.  This is similar to results I’ve blogged on elsewhere.

However, what is interesting on this study is that on multiple choice quizzes, there is the potential danger that students will choose the wrong answers and so think they have retrieved information which is in fact wrong. What this study showed was that if you give feedback on the quiz, then this improves learning further, as you can see in the graph above. Interestingly, although you might think that immediate feedback right after the question is best, this wasn’t the case in this example. Quizzes with feedback delayed until the end of the assessment gave better results than those with feedback after each question. The authors postulate that a slight gap in giving the feedback allows the incorrect concept to dissipate before the feedback is given and also gives spacing in time, which helps learning.

My summary of understanding from this research:

  • Giving a quiz after learning will help retention, as it gives recall practice
  • Giving feedback helps improve retention, particularly in multiple choice quizzes where there is a danger of learners choosing wrong answers and thinking they are right
  • Feedback is better at the end of the quiz, not after each question

For more information on the research, see Professor Roediger’s publications page at http://psych.wustl.edu/memory/publications/.

One interesting issue this raises is that it’s common in certification exams not to give feedback, to retain the confidentiality of the questions by not repeating them, and because certification aims at measuring rather than learning. What this research shows is that if you want to help your successful and failing candidates learn, then you could consider feedback in some form.

Here’s a question to allow you to practice retrieval on the subject of this blog:

Should you give feedback on multiple choice quizzes after each question or at the end of the assessment?

Answering questions directly helps you learn

Posted by John Kleeman

One of the most interesting things I’ve ever learned is that answering questions directly helps you learn. Taking a quiz or test gives your brain retrieval practice, and so helps you retrieve it again when you need it.

Smart people over the ages have realized that when you learn, you need to learn to retrieve something, not to simply store it in memory. The famous writer Francis Bacon wrote in 1620

If you read anything over twenty times, you will not learn it by heart so easily as if you were to read it only ten, trying to repeat it between whiles, and when memory failed, looking at the book.

I first learned this in an assessment context from an excellent presentation by Dr. Will Thalheimer at the first ever Questionmark user conference in Miami in 2003. He spoke about the Learning Benefits of Questions (his white paper on this subject is still available at Questionmark’s website).  But it’s exciting to see that academic research is continuing in this area since 2003, and that psychologists are getting a better understanding of how tests directly help you learn. One of the leading researchers in the field is Professor Henry L. Roediger III at the Memory Lab in Washington University in St. Louis – you can see his list of publications at http://psych.wustl.edu/memory/publications/.

Roediger and Butler (in a powerful paper on the subject) describe two ways in which quizzes and tests help learning

  • Mediated (or indirect) effects where questioning indirectly impact learning

Examples of mediated effects are that students who know they are to be tested will study for the test and so learn; that quizzes give feedback which helps learning; and that assessments can direct learners and instructors to identify weak areas where more learning is needed. These are the effects that we are all usually familiar with.

  • Unmediated (or direct) effects where questioning directly helps learning.

This is the effect whereby actually answering questions gives you retrieval practice and makes things you answer easier to recall later. When you learn something, you are not seeking just to implant it in your brain, but also to be able to retrieve it a later date. Answering a question gives you retrieval practice, and helps you be able to retrieve it later on when you need it.

Elementary or primary school teachers know the unmediated value of questioning, that’s why small children do drill and practice exercises and repeatedly are asked questions on number tables and other simple stuff.

But as learners get older, students and teachers sometimes rebel against assessments and think that learning does not benefit from them. However the scientific evidence is clear that if you study something and take a quiz or test, you retain it better than if you just study. For an instance of this, see my earlier blog entry which includes the chart below from another paper. This chart shows that if you want to learn something and retrieve it for a few minutes, lots of studying works well, but if you want to learn it and be able to retrieve it a week later, studying and testing gives much better results than just studying.


I’ll share more of my understanding on unmediated effects of questioning in future blog articles.

In the meantime, to practice what I preach, here are three questions for you to help you retain the information above.