Fair. Valid. Reliable. — Item Writing Guide, Part 1

Doug Peterson HeadshotPosted By Doug Peterson

While many people tend to think of quizzes, tests and exams as being made up of questions, professionals in the assessment industry typically use the term “item” because the question is only a part of the overall interaction with the learner, along with choices, scoring and feedback.

Well-written items can be used to assess what a learner needs to learn prior to a learning event as well as what they *have* learned after the learning event. Well-written items promote learning and memory recall, and help retain knowledge, skills and/or abilities over time. But writing good items isn’t as easy as it looks.

I’ll be devoting a few blog posts to some pointers about item writing and hope you find them helpful.

Today, let’s consider these three important qualities of well-written items – they need to be fair, valid and reliable.

Fair

“Trick” items, or confusing or misleading items do not allow the test-taker to show their true understanding of the subject matter, and stakeholders in the testing process would not be able to trust the results. The goal of an item is not to prove what a test-taker doesn’t know, it’s to allow him or her to show what they *do* know. Also, an item should only test one thing, so you only want to test the participant’s knowledge OR their puzzle-solving abilities, but not both at the same time.

Valid

An item needs to be valid in the context of the assessment. For example, an assessment that tests a learner’s ability to diagnose Alzheimer’s disease should not contain an item about dental hygiene. The dental hygiene item, as well-written as it may be, is simply not valid in that context.

Reliable

Finally, an item needs to be reliable. For example, if you were asking a five-year-old a question that requires the language comprehension skills of a ten-year-old, it’s unlikely that your results would be very reliable. An item must accurately measure the test-taker’s true understanding of the subject matter, repeatedly over time, and reduce the possibility of guessing the correct answer as much as possible.

Please feel free to leave a comment in the area below! In Part 2, we’ll take a look at the different parts of an item as well as some basic guidelines for writing a good stimulus and good choices.

Other posts in this series:

3 Responses to “Fair. Valid. Reliable. — Item Writing Guide, Part 1”

  1. Actually, in the reliable example with the five-year-old you might have a very reliable item. The child might consistently answer “I don’t know.” A thermometer can be reliable, consistent, but consistently wrong if it measures water boiling at 85 degrees F.

  2. […] you miss part 1 of this series? Click here for my first post, which focuses on the importance of making test items fair, valid and […]

  3. […] part 1 of this series we looked at the importance of fairness, validity and reliability in assessment […]

Leave a Reply