Posted by John Kleeman
Last week’s Questionmark Conference in beautiful Santa Fe was a great opportunity to meet and learn from Questionmark customers and some of the world’s leading experts in online assessment. Here are six tips I heard which I hope will interest Questionmark blog readers.
- Tip 1. Set up topic structures that are useful to report on. Define topic structures at the lowest probable reporting level so you can use topic scores to look at performance by topic in an actionable way. Make topic names meaningful to the business as elements of knowledge or skill or competency. For example, it’s often worth looking at average topic scores across a cohort of participants. If one or two topic areas are weaker than others, this likely shows an area that people do not understand or where your training is weak. In general, when designing assessments, the more you think about the decisions you will make as a result of your assessments, the more valuable your assessments will be.
- Tip 2. Get SMEs to evaluate questions checking task performance before using them. If your questions are being used to check task or job performance, you need to make sure they actually do so. One presenter explained how they send out a survey to SMEs (subject matter experts), asking about the quality and essentiality of each proposed question on a 4-point scale. The quality question asks whether the question is written well enough to accurately assess knowledge. The essentiality question asks how important the question is to meet assessment objectives. Questions with low quality should be modified or rewritten, and questions should only be included in the assessment if they score well in essentiality.
- Tip 3. If using observational assessments, keep a close watch on observer reliability. A lot of organizations use Questionmark to deliver observational assessments, which rely on a person grading a participant on performance. To make this most effective, if you have many observers/graders, put in effort to ensure all graders grade similarly (i.e. improve inter-rater reliability). If you don’t do this, there is a risk that scores will vary unfairly due to different ratings by different graders. To do this, first define the rubric or scoring rules very clearly; second, give good training to graders, with practice sessions and exemplars; and finally, monitor the grading in as close to real time as possible to catch any drift.
- Tip 4. Randomization helps security. It’s definitely helpful to shuffle choices in most questions and shuffle question order in most assessments. It’s also common to select questions from an item bank at random. From a security perspective, this makes it harder for participants to collude or pass information to one another. Also, if a participant does leak questions, the fact that each sees a randomly different test can sometimes identify the leaker due to the unique randomization he/she received.
- Tip 5. Balance your retake policy against your item bank size. If you select at random from an item bank and allow participants to retake assessments, it’s important to have a large enough item bank size to make items different each time someone retakes. A small item bank is a security weakness as it makes content exposure more damaging. It’s helpful to use the Questionmark setting that ensures that when selecting questions at random, when enough questions are available, you avoid selecting those previously delivered to the same participant.
- Tip 6: Enter into signed agreements with authors and administrators. There is a big benefit in signing formal, written security agreements with authors and administrators. This communicates very clearly that assessments and related content need to be kept confidential. Sometimes people inadvertently or deliberately leak data or let it escape and say that they did not understand the consequences; a security agreement that a person has signed makes such an excuse much harder to make and will encourage security.
Thank you to everyone who attended the conference and especially to those who shared good practice.
As a final point, I loved this motto (adapted from the famous Socrates saying) courtesy of Ted Stille at the US State Department, Diplomatic Security Service:
“The unexamined course is not worth giving”.
This might have several meanings, but the two that strike me are that any training needs an exam or other measurement to make it worthwhile, but also that you should think hard and evaluate well any training activity to ensure that your time and the time of your participants are well spent.
After I shared this lovely motto on Twitter, I was reminded by testing guru Bill Coscarelli of one of his maxims:
” You don’t get what you want, you get what you test.”
I hope you enjoyed these tips. More best practice can be found in our white papers and eBooks at: www.questionmark.com/whitepapers