The key to reliability and validity is authoring
Posted by John Kleeman
In my earlier post I explained how reliability and validity are the keys to trustable assessments results. A reliable assessment means that it is consistent and a valid assessment means that it measures what you need it to measure.
The key to validity and reliability starts with the authoring process. If you do not have a repeatable, defensible process for authoring questions and assessments, then however good the other parts of your process are, you will not have valid and reliable assessments.
The critical value that Questionmark brings is its structured authoring processes, which enable effective planning, authoring, and reviewing of questions and assessments and makes them more likely to be valid.
Questionmark’s white paper “Assessment Results You Can Trust” suggests 18 key authoring measures for making trustable assessments – here are three of the most important.
Organize items in an item bank with topic structure
There are huge benefits to using an assessment management system with an item bank that structures items by hierarchical topics as this facilitates:
- An easy management view of all items and assessments under development
- Mapping of topics to relevant organizational areas of importance
- Clear references from items to topics
- Use of the same item in multiple assessments
- Simple addition of new items within a topic
- Easy retiring of items when they are no longer needed
- Version history maintained for legal defensibility
- Search capabilities to identify questions that need updating when laws change or a product is retired
Some stand alone e-Learning creation tools and some LMSs do not provide you with an item bank and require you to insert questions individually within an assessment. If you only have a handful of assessments or you rarely need to update assessments, such systems can work, but for anyone with more than a few assessments, you need an item bank to be able to make effective assessments.
Authoring tool subject matter experts can use directly
One of the critical factors in making successful items is to get effective input from subject matter experts (SMEs), as they are usually more knowledgeable and better able to construct and review questions than learning technology specialists or general trainers.
If you can use a system like Questionmark Live to harvest or “crowdsource” items from SMEs and have learning or assessment specialists review them, your items will be of better quality.
Easy collaboration for item reviewers to help make items more valid
Items will be more valid if they have been properly reviewed. They will also be more defensible if the past changes are auditable. A track-changes capability, like that shown in the example screenshot below, is invaluable to aid the review process. It allows authors to see what changes are being proposed and to check they make sense.
These three capabilities – having an item bank, having an authoring tools SMEs can access directly and allowing easy collaboration with “track changes” are critical for obtaining reliable and valid, and therefore trustable assessments.
For more information on how to make trustable assessments, see our white paper “Assessment Results You can Trust”