The key to reliability and validity is authoring

John Kleeman HeadshotPosted by John Kleeman

In my earlier post I explained how reliability and validity are the keys to trustable assessments results. A reliable assessment means that it is consistent and a valid assessment means that it measures what you need it to measure.

The key to validity and reliability starts with the authoring process. If you do not have a repeatable, defensible process for authoring questions and assessments, then however good the other parts of your process are, you will not have valid and reliable assessments.

The critical value that Questionmark brings is its structured authoring processes, which enable effective planning, authoring, Questionmark Liveand reviewing of questions and assessments and makes them more likely to be valid.

Questionmark’s white paper “Assessment Results You Can Trust” suggests 18 key authoring measures for making trustable assessments – here are three of the most important.

Organize items in an item bank with topic structure

There are huge benefits to using an assessment management system with an item bank that structures items by hierarchical topics as this facilitates:

  • An easy management view of all items and assessments under development
  • Mapping of topics to relevant organizational areas of importance
  • Clear references from items to topics
  • Use of the same item in multiple assessments
  • Simple addition of new items within a topic
  • Easy retiring of items when they are no longer needed
  • Version history maintained for legal defensibility
  • Search capabilities to identify questions that need updating when laws change or a product is retired

Some stand alone e-Learning creation tools and some LMSs do not provide you with an item bank and require you to insert questions individually within an assessment. If you only have a handful of assessments or you rarely need to update assessments, such systems can work, but for anyone with more than a few assessments, you need an item bank to be able to make effective assessments.

Authoring tool subject matter experts can use directly

One of the critical factors in making successful items is to get effective input from subject matter experts (SMEs), as they are usually more knowledgeable and better able to construct and review questions than learning technology specialists or general trainers.

If you can use a system like Questionmark Live to harvest or “crowdsource” items from SMEs and have learning or assessment specialists review them, your items will be of better quality.

Easy collaboration for item reviewers to help make items more valid

Items will be more valid if they have been properly reviewed. They will also be more defensible if the past changes are auditable. A track-changes capability, like that shown in the example screenshot below, is invaluable to aid the review process. It allows authors to see what changes are being proposed and to check they make sense.

Screenshot of track changes functionality in Questionmark Live

These three capabilities – having an item bank, having an authoring tools SMEs can access directly and allowing easy collaboration with “track changes” are critical for obtaining reliable and valid, and therefore trustable assessments.

For more information on how to make trustable assessments, see our white paper “Assessment Results You can Trust” 

Video: Secure and Collaborative Assessment Authoring

Doug Peterson HeadshotPosted By Doug Peterson
Questionmark Live is Questionmark’s easy-to-use online item and assessment authoring tool that makes it easy for authors to collaborate securely.
In this video, we’ll take a quick look at sharing topics in Questionmark Live.

New Features Available in Questionmark Live


Posted by Jim Farrell

The Questionmark Live team has been busy working on features that enhance our powerful authoring tool. The following is a list of the new features now available in Questionmark Live:

  • New, realigned question buttons – We have aligned the question types down the left side of the screen to improve usability.
  • Cut, copy and paste buttons are now available on the question editor toolbar.
  • RSS reader to the Welcome screen – We have created a special RSS feed, with news specific to Questionmark Live, next to the Welcome screen. You can also subscribe to the feed with whatever RSS reader  you use. The Questionmark Live RSS reader also follows the Questionmark Blog.
  • Ability to add ALT tags to images – You can now add ALT tags to your images when you upload them into Questionmark Live.
  • Changes to Email Qpack – To improve deliverability, we have changed the address that sends qpacks to Perception users. Multimedia files – You can now add multimedia resources to your questions including video, audio and flash. These resources are packaged and sent with the questions when you export your qpack.

Give Questionmark Live a try, and please send your Questionmark Live feedback to