6 Steps to Authoring Trustworthy Assessments

AprilPosted by April Barnum

I recently met with customers and the topic of authoring trustworthy assessments and getting back trustable results was a top concern. No matter what they were assessing on, everyone wants results that are trustable, meaning that they are both valid and reliable. The reasons were similar, with the top three being: Safety concerns, being able to assert job competency, and regulatory compliance. I often share this white paper: 5 steps to better tests, as a strong resource to help you plan a strong assessment, and I encourage you to check it out. But here are six authoring steps to that can help you achieve trustworthy assessment results:

  1. Planning the assessment or blueprinting it. You basically are working out what it is that the test covers.
  2. Authoring or creating the items.
  3. Assembling the assessment or harvesting the items and assemble them for use in a test.
  4. Piloting and reviewing the assessment prior to using it for production use.
  5. Delivering the assessment or making the assessment available to participants; following security, proctoring and other requirements set out in the planning stage.
  6. Analyzing the results of the assessment or looking at the results and sharing them with stakeholders. This step also involves using the data to weed out any problem items or other issues that might be uncovered.

Each step contributes to the next, and useful analysis of the results is only possible if every previous stage has been done effectively. In future posts, I will go into each step in detail and highlight aspects you should be considering at each stage of the process.

assessment plan

Problems and Fixes — Item Writing Guide, Part 4

Doug Peterson Headshot Posted By Doug Peterson

In part 3 of this series on item writing, we began taking a look at some “problem questions” to figure out what was wrong with them and how to make them better. Let’s continue doing that.iwg 1

This is the ol’ “grammar give-away” problem. The stimulus ends in “a”, indicating that the answer begins with a consonant (or at least * should* begin with a consonant, if the assessment author is following standard rules of grammar). There’s only one choice that begins with a consonant, so the participant doesn’t need to know the answer – they just need to know a little grammar.

There are a couple of ways to fix this. One would be to end the stimulus with “a/an”. Another way would be to move the indefinite article (yes, I had to look that up) into the choice: an apple, a banana, an orange, and an eggplant.

Also be sure not to mix a singular in the stimulus with plurals in the choices, or vice versa. And if you’re writing questions in gender-specific languages like Spanish, French, or Italian, be sure to account for masculine and feminine definite and indefinite articles.

This question has a couple of things wrong with it:

iwg 2The first problem is pretty obvious. One choice is significantly longer than the other three. Typically this would mean that choice (b) is the correct answer, and in this case, that would be true.

Can you spot the other problem? It’s a little more subtle. The stimulus uses an important word – “strings” – and only the correct answer uses this word (in its singular form) as well. Without knowing anything about bass guitars, most people would answer this question correctly simply by noticing the use of the same important word in both the stimulus and one of the choices.

To fix this question, the second choice should be changed to something like “Set the intonation.” At that point the length of the correct choice is about the same as the length of the other choices, and the important word “string(s)” is not being used.

Please feel free to add your comments to this discussion – the more, the merrier! In our next installment, we’ll diagnose two more problems, and then wrap things up with a little summary.