Get trustable results : Let participants comment on beta questions
Posted by John Kleeman
However good our item creation and review processes are, we need to pilot them before we use them. As my colleague Austin Fossey says in his blog post Field Test Studies: Taking your items for a test drive:
Even though we work so hard to write high-quality items, some bad items may slip past our review committees. To be safe, most large-scale assessment programs will try out their items with a field test.
For smaller and medium-sized programs, who inevitably have less resources, field testing items is vital. In his post, Austin describes great way to do this with Questionmark – to set questions as “experimental” and include them within production assessments. Such questions gather statistics but don’t count towards the pass/fail of the participant.
Another useful capability in Questionmark technology that helps with field testing or beta testing of questions is allowing participants to comment on a question. This allows you to place an optional text box underneath a question, and participants can enter a comment if they feel a question is poorly worded or ambiguous.
It’s easy to turn on a comment box in Questionmark Live — you just click on the Add comment box button highlighted in the screenshot below. You can also easily do this in our other authoring tools.
Participants can then enter comments, which appear in reports and can easily be collated and actioned. When you finalize the questions to use in production, you can simply turn the comments off.
I hope that getting feedback and the ability to improve questions before they are used in production help you create more trustable assessments. By following these steps, you also demonstrate due diligence by trying out items before you use them for scoring, thereby minimizing the risk of having your assessment challenged later.
If you have any comments on this Questionmark capability or how best to field test items, feel free to enter them in the comment area below!
Check out our white paper: 5 Steps to Better Tests for best practice guidance and practical advice for the five key stages of test and exam development.
John Kleeman will discuss benefits and good practice in assessments at the 2015 Users Conference in Napa Valley, March 10-13. Register before Dec. 17 and save $200.