Seven deadly sins of item writing – with apologies to William Shakespeare
Posted by John Kleeman
“Double, double toil and trouble;
Fire burn, and caldron bubble.”
So said the three witches in Shakespeare’s Macbeth as they prepared to lead the lord Macbeth to doom and destruction.
Today is Halloween and the evenings in London have been getting darker. A few days ago, I was walking home through Hampstead Heath after work. The wind whispered through the trees, and there was thunder, lightning and rain in the air. I thought I heard witches’ voices in the distance. Was it a dream or was it real? I do not know, but here is what I can recall:
The witches were discussing how to muddle, befuddle or scuttle a testing programme. They suggested seven deadly sins of item writing:
1. Use lots of not. Don’t avoid negatives: indeed if it’s not too hard, include not a few double negatives not infrequently to make your test-takers struggle.
2. Trick your test-takers. Remember how we witches tricked Macbeth to murder the King and showed up his weaker qualities. Mislead your test-takers, add tricks into your questions, so only the truly strong will pass your tests.
3. Include speling and grammer errors. It’ll make your test takers feel more at ease if you include a few language misteaks in each question.
4. Never use one word where you can use 13. Make your question wording and choices as long as you can; that will sort out the toads from the frogs.
5. We love “All of the above” and “None of the above”. They give so much room for confusion, delusion and illusion in multiple choice items.
6. Subtle, subtle is number six. Make sure your items test factual knowledge only; don’t test higher order skills like comprehension or application. Avoid scenarios or anything that checks real-world skills; measure as little useful capability as you can.
7. Most importantly, skip all the review. Having other people read and review your questions, fit them to a blueprint, check them for language, psychometrics and bias is so boring! Be bloody, bold and resolute – deploy your questions without wasting time waiting for others’ opinions.
This is fiction of course. But fair is foul, and foul is fair – follow these seven rules if you want your tests to be unfair!