Conference Close-up: Timing is Everything

Posted by Joan Phaup

As many followers of this blog will know, Questionmark Chairman John Kleeman has been exploring the findings of cognitive learning research and considering how it can apply to assessments.

At this year’s Questionmark Users Conference in New Orleans, John will explain research on the value of spacing out learning and assessment as means of helping people remember information for the long term.

I asked him the other day about his presentation, Timing is Everything : Using psychology research to make your assessments more effective”

What do you value you most about what you’ve been learning from the work of cognitive learning researchers?

John Kleeman

There’s a lot of evidence from cognitive psychology that could make a difference in how we do assessments, training and learning, and I want people to be aware of it. I’m excited about research that’s being done and am keen to share research findings so that people involved in learning and assessment can use that evidence in practical ways.

What key findings about the timing of assessments will you share during your breakout session?

There’s a fascinating and well-documented finding that if you space out learning – separate it out – it’s much more effective than if you do it all in one chunk. Learning should be regarded as a process, not an event. Research shows that if you spend half an hour a day for four days learning something it will be more effective than if you cover all the material in a single two-hour session. Because of the way the mind works, having breaks between learning sessions will help you remember information for the long term. I’ll be sharing solid evidence about this and will talk about its implications.

Assessment plays into all this! An assessment with feedback is learning, and so if you take a series of separated-out assessments, this gives you spacing. Also if you give learners a series of assessments during a course, and encourage them to learn and revise for each assessment, you are encouraging this good behavior of spacing out their learning. So if you have people take quizzes and tests throughout learning instead of just at the end, you are promoting some good learning habits.

Can you give me an example of this?

I wrote not too long ago about how the University of Lund in Sweden uses embedded assessments for knowledge checks within a SharePoint-based learning platform. They’ve found that requiring students to take quizzes as they work through distance learning courses forces them to engage with the material and practice retrieving information from the very start of their courses — and to keep that engagement going throughout the course. That’s a great way to prevent people from cramming a lot of learning into a short period of time – as they might do before a final exam. So assessments can be used to help space out learning. If you just rely on final tests or exams then there is a risk of encouraging people to cram at the end of the course, rather than helping them learn and remember information for the long term.

How can we use research findings about the spacing of learning in designing tests and quizzes?

One of the things I covered last year and will review in this session is the benefit of retrieval practice – the fact that if you want to remember something for the long term then you want to practice retrieving it from memory. Things you have practiced retrieving are easier to remember. So if you have to answer a question or take a test on material, it makes it more likely you will remember it in future. For example, for retrieval practice it’s best if possible to use open-ended questions that require the person to recall information rather than recognize it from a multiple choice list. And you definitely should include feedback.

Another point to keep in mind is that the material that gets tested is the material people will remember, so it’s important to cover the material you really regard as valuable and worth remembering. We’ll discuss these and other ideas about how to apply the research to assessments during the session.

What you do expect participants in your session to take away from it?

I’ll be sharing some of the evidence from cognitive psychology both to communicate to what I understand to be the results and to have people come up with their own views. I will also share links to resources from experts like Dr. Will Thalheimer of Work-Learning Research so that they can learn more on their own.

I don’t just want to say, “Here’s the data and believe me!” I want to give attendees a way to look into this themselves and understand it. Once they grasp the principles coming out of the research, it will help them formulate assessment practices and plans in ways that apply to their individual organizations’ needs.

We hope to see you in New Orleans March 20 – 23 and encourage you to sign up by January 27 for early-bird savings.

Meaningful Feedback: Some good learning resources


Posted by Jim Farrell

December is the time to take stock of the year that’s winding down, and a highlight for me in 2010 was attending the eLearning Guild’s DevLearn conference. One of the things I enjoy most about DevLearn is attending the general sessions where industry leaders speak passionately about the state of elearning and  important trends like social networking, games and simulations in learning.

One of the speakers at this year’s closing session  was Dr. Jane Bozarth, the elearning coordinator for the North Carolina Office of State Personnel. Jane is a great person to follow on Twitter (and not just because she is a fellow resident of the triangle here in NC). Jane’s tweets are full of valuable resources, and one of the many topics that interests her (and me!) is the use of feedback in learning and assessments. Jane’s recent article on Nuts and Bolts: Useful Interactions and Meaningful Feedback in Learning Solutions Magazine includes some great examples of feedback. In that article,  Jane emphasizes that “the point of instruction is to “support gain, not expose inadequacy” — and that feedback should be provided with that goal in mind.

Jane’s article reminded me that during one of our Questionmark Podcasts, Dr. Will Thalheimer of Work-Learning Research notes the importance of retrieval practice in the learning process and the role of feedback in supporting retrieval. The amount of feedback is tied to when the assessment comes in the learning process. For instance,  feedback with a formative assessment can pave new paths to information that can make future retrieval easier. Feedback for incorrect responses during learning is used to repair misconceptions and replace them with correct information and a new mental model that will be used to retrieve information in the future. As Dr. Thalheimer mentions in the podcast, good authentic questions that support retrieval also support good feedback. You will find more details about this in Dr. Thalheimer’s research paper, Providing Feedback to Learners, which you can download from our Web site.

All these resources can help you use feedback to “support gain, not expose inadequacy,” making your assessments in the coming year more effective.

A conversation on the value of asking good questions

Joan Phaup

Posted by Joan Phaup

I enjoyed  a blog post by Andy Klee of  Klee Associates about a  recent conversation he’d had with our own John Kleeman, Questionmark’s chairman. Klee, whose organization provides JD Edwards and SAP training and consulting, showed a great deal of interest in how good questions and tests can improve learning outcomes.

Click here to follow their wide-ranging discussion, which covers such topics as the challenge of creating high-quality test questions, the correlation between performance on certification exams and  future job performance, and trends in exam design and administration.

It’s great to see more and more people recognizing that asking questions adds value to learning. If you would like to read more on this subject, check out this paper by Dr Will Thalheimer of Work-Learning Research: The Learning Benefits of Questions.

Answering questions directly helps you learn

Posted by John Kleeman

One of the most interesting things I’ve ever learned is that answering questions directly helps you learn. Taking a quiz or test gives your brain retrieval practice, and so helps you retrieve it again when you need it.

Smart people over the ages have realized that when you learn, you need to learn to retrieve something, not to simply store it in memory. The famous writer Francis Bacon wrote in 1620

If you read anything over twenty times, you will not learn it by heart so easily as if you were to read it only ten, trying to repeat it between whiles, and when memory failed, looking at the book.

I first learned this in an assessment context from an excellent presentation by Dr. Will Thalheimer at the first ever Questionmark user conference in Miami in 2003. He spoke about the Learning Benefits of Questions (his white paper on this subject is still available at Questionmark’s website).  But it’s exciting to see that academic research is continuing in this area since 2003, and that psychologists are getting a better understanding of how tests directly help you learn. One of the leading researchers in the field is Professor Henry L. Roediger III at the Memory Lab in Washington University in St. Louis – you can see his list of publications at

Roediger and Butler (in a powerful paper on the subject) describe two ways in which quizzes and tests help learning

  • Mediated (or indirect) effects where questioning indirectly impact learning

Examples of mediated effects are that students who know they are to be tested will study for the test and so learn; that quizzes give feedback which helps learning; and that assessments can direct learners and instructors to identify weak areas where more learning is needed. These are the effects that we are all usually familiar with.

  • Unmediated (or direct) effects where questioning directly helps learning.

This is the effect whereby actually answering questions gives you retrieval practice and makes things you answer easier to recall later. When you learn something, you are not seeking just to implant it in your brain, but also to be able to retrieve it a later date. Answering a question gives you retrieval practice, and helps you be able to retrieve it later on when you need it.

Elementary or primary school teachers know the unmediated value of questioning, that’s why small children do drill and practice exercises and repeatedly are asked questions on number tables and other simple stuff.

But as learners get older, students and teachers sometimes rebel against assessments and think that learning does not benefit from them. However the scientific evidence is clear that if you study something and take a quiz or test, you retain it better than if you just study. For an instance of this, see my earlier blog entry which includes the chart below from another paper. This chart shows that if you want to learn something and retrieve it for a few minutes, lots of studying works well, but if you want to learn it and be able to retrieve it a week later, studying and testing gives much better results than just studying.


I’ll share more of my understanding on unmediated effects of questioning in future blog articles.

In the meantime, to practice what I preach, here are three questions for you to help you retain the information above.

New York Times advises that tests help you retain learning

john_smallPosted by John Kleeman

I’d like to draw your attention to a thought-provoking article in the New York Times earlier this week about the best way to  learn.

One interesting observation in the article is that although you might think going and staying at a quiet place to study is the best way to learn, this isn’t the case. It’s actually easier to learn if you move around to different places! It would seem that when the outside context varies, it’s easier to put on the neural scaffolding that helps retain something in memory.

And, mirroring papers by Dr. Will Thalheimer commissioned by Questionmark (see The Learning Benefits of Questions and Providing Learners with Feedback), tests also help the retention of learning. In particular the New York Times describes an experiment at Washington University in St. Louis where two sets of students studied a reading passage in different ways. One set studied it twice in back-to-back sessions, the other set studied it once and then took a practice test on it, within the same time. As you can see in the diagram below, students who studied only learned the information well at the time, but forget about half of it within a week. But those who studied and had a practice test, retained much more of the information.

Chart from University of Washington research paper, click to read paper

The bottom line from the research is that taking memory tests improves long-term retention. Tests don’t just measure learning: the act of taking a test helps you retain information you have learned.