Test above knowledge: Use scenario questions

John Kleeman portraitPosted by John Kleeman

Here’s the one piece of advice I’d give above all others to anyone creating quizzes, tests or exams: Test above knowledge.

You may be familiar with Bloom’s taxonomy of learning objectives, which is shown in the diagram below. At the lowest level is Knowledge; questions that test Knowledge ask for simple remembering of facts. At the highest level is Evaluation; questions that test Evaluation require participants to use judgement.

Bloom pyramid

It’s great if you can write questions that assess at the highest levels of Bloom’s Taxonomy, but many organizations have a lot to gain by asking questions at any of the levels  above Knowledge in the pyramid. Although there are times when testing facts can be useful, it’s usually better to test the application or comprehension of those facts.

In today’s world, where facts are easily googleable, it is the use and understanding of those facts in the real world that is genuinely useful. By testing above knowledge, you are checking not just that the participant knows something but that they can apply it in some scenario. This is more valid and more realistic — and for most applications it is also more useful.

Here is a simple example to illustrate the point:

Traffic lightsWhat does a yellow traffic light mean?

  • Stop
  • Go
  • Caution

This is purely a factual, knowledge question.

But here, the question requires that the respondent to apply to meaning of a yellow traffic light to an actual situation:

If you are driving toward an intersection and the light turns from yellow to red, what should you do?

  • Speed up and cross the intersection
  • Stop suddenly
  • Stop gradually

This is a very simple example, but I hope it makes you realize that converting factual questions to scenarios is not very hard.

I’d encourage you to consider using scenarios in your questions: Ask people to apply their knowledge, not just prove that they know some facts. Have your test-takers apply what they know to actual situations.

SlideShare presentation on writing high-complexity test items

Headshot JuliePosted by Julie Delazyn

Writing high-quality test items is difficult, but writing questions that go beyond checking knowledge is even more complex.

James Parry, E-Testing Manager at the U.S. Coast Guard Training Center in Yorktown, Virginia, offered some valuable tips on advanced test item construction during a peer discussion at this year’s Questionmark Users Conference.

The PowerPoints from this session will help you distinguish among three levels of test items:

  • Low-complexity – requiring knowledge of single facts
  • Medium-complexity – requiring test takers to know or derive multiple facts
  • High-complexity – requiring test takers to analyze and evaluate multiple facts to solve problems (often presented as scenarios)

The slides relate these levels to Bloom’s Taxonomy and Gagne’s Nine Events of Instruction and offer pointers for writing performance-based test items based on clear objectives.

Enjoy the presentation below, and save March 4 – 7 next year for the Questionmark 2014 Users Conference in San Antonio, Texas.