Why a “good enough” assessment solution will do more harm than good

Posted by Chloe Mendonca

Regardless of the size of a department or organization, we’re all keen to improve efficiency and when it comes to assessments — improve validity and reliability. Selecting the right assessment tool is critical. Sticking with a “good enough” solution will do more harm than good. In the short term it will hinder your productivity; in the long run it could not only mean a painful migration from one system to another, but could also cost you a fortune in compliance fines or HR disputes!

“Good enough” assessment solutions are usually fine for creating basic surveys and quizzes, but when you use assessment results to make defensible decisions about people, you need a more complete solution. So how do you distinguish a “good enough” assessment solution from a complete one?

Choosing a complete solution

As you go through the selection process, consider whether simplicity comes at the expense of security or robust functionality.We know it can be confusing to sort through the extensive number of options. And we know every company has different assessment needs. If you need to weigh up your options, this cheat sheet will highlight the limitations of a “good enough” assessment tool and explain how a complete solution will help make your job easier and your assessments more valid and reliable.

Complete assessment management platforms are designed with security, validity and reliability in mind

The cheat sheet also discusses how complete assessment management technologies trump “good enough” solutions in several areas, including:

  • Establishing content validity
  • Managing multilingual assessments
  • Maintaining an audit trail
  • Developing observational assessments
  • Minimizing cheating and content theft

It’s worth spending a good amount of time fully evaluating your options and choosing a solution that can scale with your organization. Interested in learning more about how to evaluate a complete assessment solution? Download this cheat sheet, “Are you settling for a “good enough” assessment solution?”

Questionmark Conference 2017: Three hot topics from Santa Fe

Bart Hendrickx SmallPosted by Bart Hendrickx

There was a lot to learn at the Questionmark Conference 2017, as my colleagues Chloe Mendonça and John Kleeman alluded to in their blog posts Reminiscing about Santa Fe and Assessment good practice: 6 tips from Santa Fe. I had the privilege to present several sessions and interact with a lot of customers. Here are the key things I took away from the conference:

People want to brand assessment delivery screens

This is not new: we have been supporting the branding and customization of assessment delivery screens for many years. The new Questionmark OnDemand Portal also offers various options  for branding the user experience, including for admin users – a new development compared to the classic Portal. However, most people are especially interested in customizing the way assessments are delivered to participants.

Roles and permissions can be a little daunting

The new Questionmark Portal offers many capabilities relating to roles and permissions. Refer to these blog posts from last year to get an idea: Role-based permissions: A how-to guide part 1 and part 2. While people appreciate the power of Questionmark OnDemand’s roles and permission management, they also understand that sometimes it requires a specific combination of permissions to get a role “just right”. We received several suggestions from customers on how to make this part of the software more welcoming to new users.

Integrations with Learning Management Systems remain important

Again, not something new but it was interesting to experience just how important LMS integrations are for our users. Questionmark has been supporting several integration standards for a long time. We have also enhanced the ubiquitous SCORM integrations by making it possible to include SAML in the workflow. This enables customers to combine SCORM’s launch and track capabilities with SAML’s security and demographic mapping, resulting in more secure assessment delivery and additional participant demographics that can be used in reports.

I would like to thank everyone who shared such useful feedback with us.

Want to enhance exam integrity? Need to ensure compliance? Then don’t miss these webinars!

Posted by Chloe Mendonca

Proxy testing … Cheating …. Content theft. Does the battle ever end? The techniques and technologies being used to compromise the integrity of your test programs are complex and continually evolving. Many organisations are increasingly turning to online proctoring solutions.

Research shows that when properly applied, online proctoring can be as or more effective than traditional test centre proctoring. Besides reducing security risks, it provides test takers the convenience of taking tests from their homes, workplaces or colleges. It also reduces costs not only for administrators but also for test takers. Consider for a moment the testing fees, travel, parking, and the simple time away from work or studies.  In addition to online proctoring’s effectiveness one of the most exciting things is that this approach has the potential to grow as technology becomes more advanced and widespread.

Before implementing online proctoring, organisations should approach with caution, not all online proctoring services are equivalent. While some are designed with high-stakes tests in mind others can actually increase the risk of cheating. So what do you need to consider? What should you be wary of? Is online proctoring suited to your organisation or institution? Join Questionmark for a 45-minute webinar answering these questions and more.

Register for: Enhancing Exam Integrity with Online Proctoring

Online Proctoring is only a small part of achieving exam integrity. For those less familiar with Questionmark’s end-to-end Assessment Management System, find out how security is embedded at every stage of assessment development in our 60-minute demo. Discover the features and functions available to help you securely create, deliver and report on your surveys, quizzes, tests and exams in our introductory webinar. Note, we are also holding this webinar in Dutch.

Register for: An Introduction to Questionmark’s Assessment Management System

Or for organisations working in sectors that demand compliance, learn about 7 ways that assessments can enhance your eLearning and compliance strategy and best practices for using them in our 30-minute German webinar.

Register for: Warum und wie Sie Compliance-Prüfungen nutzen sollten


Questionmark OnDemand Assessment Management System now HIPAA-compliant

Jamie ArmstrongPosted by Jamie Armstrong

Questionmark recently began offering US OnDemand Service customers the option of entering into an additional agreement for compliance with HIPAA (the US Health Insurance Portability and Accountability Act).

I’d like to provide some brief information on this exciting new development, particularly for those not familiar with what HIPAA is or involves. You can easily find additional information and resources on the U.S. Department of Health & Human Services website.

What is HIPAA and what kind of information or data does it cover?
HIPAA is a US federal law that in very general terms regulates access to and handling of “protected health information” (“PHI”) and provides individuals with important rights regarding their health information. PHI includes these categories of information:

  • health information collected from a person;
  • information relating to health conditions or health care provision created or received by an organization such as a health care provider, and;
  • information that either identifies or can reasonably be used to identify an individual.

For example, data gathered or used as part of an assessment using Questionmark OnDemand that relates to past, present or future health or condition may be PHI under HIPAA.

What types of organizations are subject to HIPAA requirements?
HIPAA applies to two main categories of organization having access to PHI. These are known as “covered entities” and “business associates.” A Questionmark customer that is a health plan or health care provider, e.g. a hospital, clinic or health insurance company,  may be a covered entity for HIPAA. Business associates include organizations receiving or maintaining PHI on behalf of a covered entity for functions such as data processing or administration (among other things). Questionmark may be a business associate in providing the OnDemand Service to customers that are either covered entities or business associates performing services for their own covered-entity clients.

What does HIPAA require?
HIPAA requires that covered entities and business associates meet various security, breach notification and privacy requirements. They must meet the requirements applicable to them internally and also have contracts with any third parties that may have access to PHI. This ensures  that these third parties are subject to the same restrictions and conditions. Before offering OnDemand Service customers the option of entering into a HIPAA business associate agreement, Questionmark completed a security and legal review to ensure compliance with relevant HIPAA requirements.

We are interested in obtaining HIPAA-compliant OnDemand Services.  How do we sign a HIPAA business associate agreement with Questionmark?
You can find our HIPAA business associate agreement here. If you’d like to learn more please contact your account manager. Questionmark is committed to safeguarding PHI in accordance with the HIPAA standards and looks forward to discussing your HIPAA compliance requirements.

Important disclaimer: This blog is provided for general information and interest purposes only, is non-exhaustive and does not constitute legal advice. As such, the contents of this blog should not be relied on for any particular purpose and you should seek the advice of your own legal counsel in considering HIPAA requirements.

Reminiscing about Santa Fe: Presentations, pictures & the weird and wonderful art house

Posted by Chloe Mendonca

After eagerly looking forward to Questionmark’s most important annual learning event for months, it was over before we even knew it! The Questionmark Conference gave all of us three special days to meet so many of our globally dispersed customers and employees face to face, learn best practices, have fun with one another and discuss new ways to leverage Questionmark’s technologies.

This year I was fortunate enough to be there, and a big highlight was getting a deeper understanding of how others are using Questionmark’s technologies. From our evening networking events to our stimulating panel discussion — which brought together experts from the US State Department, Caterpillar Inc., Scantron and Compass Consultants to discuss best practices for making data work within learning and assessment programs — to more specific breakout sessions, our guest speakers did a wonderful job of sharing lessons learned and best-practice tips.

Todd Horner from Accenture, for example, hosted a great discussion, “Taking the Migraine out of Migration: Accenture’s journey to next-gen authoring.” He spoke about the shared “fear of the unknown” and how to get around change-management challenges. Lauri Buckley and Lindsey Clayton from Caterpillar Inc, delivered an impressive presentation, “A Process to Mastery: Assessments as career development tools,” during which they shared valuable tips about how to effectively design and develop various types of competence assessments, from proficiency tests to validation and observational assessments. You can get the handouts from these presentations and more right here.

For those who couldn’t be there in person, we webcast selected conference sessions — hitting record numbers online. If you joined us for the webcast, got a sense of the Questionmark Conference atmosphere and want to join us in person next year, keep your eyes peeled for our dates and location announcement coming to the blog in the next few months. See the recordings of our selected webcast sessions at: www.questionmark.com/go/2017uconwebcast (Please note you must be logged into the website with your Questionmark username and password).

I’d like to take this opportunity to say a big thank you to all of our wonderful speakers for taking the time to share their knowledge. Without them there would be no conference!

Now for the bit you’ve all been waiting for… conference pictures! To all those who went back to the office struggling to describe the weird and wonderful art house that is Meow Wolf’s House of Eternal Return, hopefully these snaps will make things a little easier 😊  View conference and evening event pictures here on our flickr page.

What did you enjoy most about Questionmark Conference 2017? Leave me a comment below and stay in touch!

Just in case you missed it…

John Kleeman, Questionmark’s Founder & Executive Director reported back 6 good practice tips heard in Santa Fe.

Assessment good practice: 6 tips from Santa Fe

John Kleeman HeadshotPosted by John Kleeman

Last week’s Questionmark Conference in beautiful Santa Fe was a great opportunity to meet and learn from Questionmark customers and some of the world’s leading experts in online assessment. Here are six tips I heard which I hope will interest Questionmark blog readers.

  • Tip 1. Set up topic structures that are useful to report on. Define topic structures at the lowest probable reporting level so you can use topic scores to look at performance by topic in an actionable way. Make topic names meaningful to the business as elements of knowledge or skill or competency. For example, it’s often worth looking at average topic scores across a cohort of participants. If one or two topic areas are weaker than others, this likely shows an area that people do not understand or where your training is weak. In general, when designing assessments, the more you think about the decisions you will make as a result of your assessments, the more valuable your assessments will be.


  • Tip 2. Get SMEs to evaluate questions checking task performance before using them. If your questions are being used to check task or job performance, you need to make sure they actually do so. One presenter explained how they send out a survey to SMEs (subject matter experts), asking about the quality and essentiality of each proposed question on a 4-point scale. The quality question asks whether the question is written well enough to accurately assess knowledge. The essentiality question asks how important the question is to meet assessment objectives. Questions with low quality should be modified or rewritten, and questions should only be included in the assessment if they score well in essentiality.


  • Tip 3. If using observational assessments, keep a close watch on observer reliability. A lot of organizations use Questionmark to deliver observational assessments, which rely on a person grading a participant on performance. To make this most effective, if you have many observers/graders, put in effort to ensure all graders grade similarly (i.e. improve inter-rater reliability). If you don’t do this, there is a risk that scores will vary unfairly due to different ratings by different graders. To do this, first define the rubric or scoring rules very clearly; second, give good training to graders, with practice sessions and exemplars; and finally, monitor the grading in as close to real time as possible to catch any drift.


  • Tip 4. Randomization helps security. It’s definitely helpful to shuffle choices in most questions and shuffle question order in most assessments. It’s also common to select questions from an item bank at random. From a security perspective, this makes it harder for participants to collude or pass information to one another. Also, if a participant does leak questions, the fact that each sees a randomly different test can sometimes identify the leaker due to the unique randomization he/she received.


  • Tip 5. Balance your retake policy against your item bank size. If you select at random from an item bank and allow participants to retake assessments, it’s important to have a large enough item bank size to make items different each time someone retakes. A small item bank is a security weakness as it makes content exposure more damaging.Dialog box that shows when you randomly select 5 questions from a topic, you can tick on "Avoid previously delivered" It’s helpful to use the Questionmark setting that ensures that when selecting questions at random, when enough questions are available, you avoid selecting those previously delivered to the same participant.


  • Tip 6: Enter into signed agreements with authors and administrators. There is a big benefit in signing  formal, written security agreements with authors and administrators. This communicates very clearly that assessments and related content need to be kept confidential. Sometimes people inadvertently or deliberately leak data or let it escape and say that they did not understand the consequences; a security agreement that a person has signed makes such an excuse much harder to make and will encourage security.

Thank you to everyone who attended the conference and especially to those who shared good practice.

As a final point,  I loved this motto (adapted from the famous Socrates saying) courtesy of Ted Stille at the US State Department, Diplomatic Security Service:

“The unexamined course is not worth giving”.

This might have several meanings, but the two that strike me are that any training needs an exam or other measurement to make it worthwhile, but also that you should think hard and evaluate well any training activity to ensure that your time and the time of your participants are well spent.

After I shared this lovely motto on Twitter, I was reminded by testing guru Bill Coscarelli of one of his maxims:

” You don’t get what you want, you get what you test.”

I hope you enjoyed these tips. More best practice can be found in our white papers and eBooks at: www.questionmark.com/whitepapers

« Previous PageNext Page »