Problem Questions and Summary – Item Writing Guide, Part 5

Doug Peterson Headshot

Posted By Doug Peterson

Let’s look at two more item writing problems. These last two are a little controversial.5- a

The stimulus for this question tells a wonderful story. The problem is, the first three sentences contain no information that relates to the question. A long stimulus full of extraneous, unneeded information can easily distract or confuse the test-taker. This item needs a re-write of the stimulus to get directly to the question at hand – and nothing else. Let’s change it to “Which Questionmark video explains how to use assessments in solving business problems?”

But here’s the controversy: This question is just fine if you’re trying to ascertain the test-taker’s ability to recognize pertinent information and ignore extraneous information! Therefore I won’t advise that you *never* use a question like this, only that you make sure you use it in the right situation.

And now, on to our last question in this series of posts.

5 -bAt first glance, there doesn’t appear to be a problem with this question – no repetition of a keyword, distracters are the same length, no grammar inconsistencies, short and to the point… But note the word “not” in the stimulus.

The other questions we’ve looked at in part 3 and part 4 of the series ask the test-taker to find the *correct* answer, but this question suddenly has them looking for an *incorrect* answer. This requires the test-taker to reverse their approach to the question, which can be very confusing.

That being said, there are some who advocate putting a certain number of negative questions on a *survey* to help ensure that the person filling it out is paying attention and not just flying through the questions. I’m not sure I agree with this approach. I feel that if they’re not interested and not paying attention to what they’re doing, negative questions aren’t going to change that, but they could lead to some very bad data being collected.

When it comes to quizzes, tests and exams, especially high-stakes exams, I strongly advise against using negative questions. If you absolutely must use a negative question, emphasize the negative by using all capital letters, bold it, and maybe even underline it.

So let’s pull it all together. It’s important to be fair to both the test-taker and the testing organization.

  • The test-taker should only be tested for the knowledge, skills or abilities in question, and nothing else.
  • The testing organization needs to be assured that the assessment accurately and reliably measures the test-taker’s knowledge, skills or abilities.

To do this, your assessment needs to be made up of well-written questions. To write good assessment questions:

  • Be careful with your wording so that you don’t create overly long or confusing questions.
  • Be concise. Sentences should be as short as possible while still posing the question clearly.
  • Keep it simple. Avoid compound sentences and use short, commonly used words whenever possible. Technical terminology is acceptable if it is part of what
    the test measures.
  • Make sure each question has a specific focus, and that you’re not actually testing multiple pieces of knowledge in a single question.
  • Always use positive phrasing to avoid confusion. If you have no choice but to use negative phrasing, make sure that the negative word – for example,
    “not” – is emphasized with capital letters, bold font, and/or underlining.
  • When creating distracters:
  • keep them all the same relative length,
  • as short as possible,
  • avoid using keywords from the stimulus,
  • watch out for grammatical cues, and
  • make sure that all distracters are reasonable answers within the context of the question.

As always, feel free to leave your comments, or contact me directly at doug.peterson@questionmark.com.

Leave a Reply