The key to reliability and validity is authoring

John Kleeman HeadshotPosted by John Kleeman

In my earlier post I explained how reliability and validity are the keys to trustable assessments results. A reliable assessment means that it is consistent and a valid assessment means that it measures what you need it to measure.

The key to validity and reliability starts with the authoring process. If you do not have a repeatable, defensible process for authoring questions and assessments, then however good the other parts of your process are, you will not have valid and reliable assessments.

The critical value that Questionmark brings is its structured authoring processes, which enable effective planning, authoring, Questionmark Liveand reviewing of questions and assessments and makes them more likely to be valid.

Questionmark’s white paper “Assessment Results You Can Trust” suggests 18 key authoring measures for making trustable assessments – here are three of the most important.

Organize items in an item bank with topic structure

There are huge benefits to using an assessment management system with an item bank that structures items by hierarchical topics as this facilitates:

  • An easy management view of all items and assessments under development
  • Mapping of topics to relevant organizational areas of importance
  • Clear references from items to topics
  • Use of the same item in multiple assessments
  • Simple addition of new items within a topic
  • Easy retiring of items when they are no longer needed
  • Version history maintained for legal defensibility
  • Search capabilities to identify questions that need updating when laws change or a product is retired

Some stand alone e-Learning creation tools and some LMSs do not provide you with an item bank and require you to insert questions individually within an assessment. If you only have a handful of assessments or you rarely need to update assessments, such systems can work, but for anyone with more than a few assessments, you need an item bank to be able to make effective assessments.

Authoring tool subject matter experts can use directly

One of the critical factors in making successful items is to get effective input from subject matter experts (SMEs), as they are usually more knowledgeable and better able to construct and review questions than learning technology specialists or general trainers.

If you can use a system like Questionmark Live to harvest or “crowdsource” items from SMEs and have learning or assessment specialists review them, your items will be of better quality.

Easy collaboration for item reviewers to help make items more valid

Items will be more valid if they have been properly reviewed. They will also be more defensible if the past changes are auditable. A track-changes capability, like that shown in the example screenshot below, is invaluable to aid the review process. It allows authors to see what changes are being proposed and to check they make sense.

Screenshot of track changes functionality in Questionmark Live

These three capabilities – having an item bank, having an authoring tools SMEs can access directly and allowing easy collaboration with “track changes” are critical for obtaining reliable and valid, and therefore trustable assessments.

For more information on how to make trustable assessments, see our white paper “Assessment Results You can Trust” 

Easier collaboration in Questionmark Live

Posted by Julie Delazyn

We always like sharing news about Questionmark Live, our browser-based assessment authoring tool, on the blog, including the recent addition of hierarchical topics. Another big change is the new interface, which makes it easier than ever for subject matter experts and test designers to collaborate on an assessment. We’ve made it very easy to share topics and subtopics.

How does it work? Simply click the share button, as you can see in the screen shot below, and type the email of the person you would like to share the subtopic with. The recipient will receive an invitation to view the topic in Questionmark Live (to see context) and the sub-topic they can work in. It’s that easy.

The subtopic folder that you shared will now display a green arrow, as shown in the screenshot below. Click on the number beneath the “No. of Revisions” column to track a question’s revision history as well as to see who edited the question and which changes were made. You can also compare the different versions of the same question and roll back to a previous version. Learn more about revision history in Questionmark Live in this blog post.

Crowdsourcing Content

jim_smallPosted by Jim Farrell

Jay Cross recently mentioned in his Learning Blog a Harvard Business blog post by John Hagel III, John Seely Brown, and Lang Davison about The Collaboration Curve.  I was struck by this statement from their article:

…the more participants–and interactions between those participants–you add to a carefully designed and nurtured environment, the more the rate of performance improvement goes up

This is why Questionmark Live is so important. Think of it as crowdsourcing content from participants who may previously  have been indifferent to the assessment process within your organization. As participation and collaboration increases, the quality of your questions will improve and become far more job-relevant. Look at the following quote from our friends at the Harvard School of Business and replace collaboration curves with Questionmark Live.

Collaboration curves hold the potential to mobilize larger and more diverse groups of participants to innovate and create new value.

Questionmark Live is a vital tool in the creation of a true learnscape.