Item Writing & Questionmark Boot Camp: Pre-Conference Workshops

Rick Ault, Questionmark Trainer

Julie ProfilePosted by Julie Delazyn

Planning for Questionmark Conference 2017 in Santa Fe, New Mexico, March 21-14 is well underway.

We have begun posting descriptions of breakout sessions and are pleased to announce two pre-conference workshops.

Both of these all-day sessions will take place on Tuesday, March 21, 2017:

assessments-2017Questionmark Boot Camp: Basic Training for Beginners

Do you want to get the most out of the Questionmark Users Conference even though you are just starting out with Questionmark?

Here’s what to do: Bring your laptop and learn directly from Questionmark expert Rick Ault.

Bring your laptop and get into gear with hands-on practice in creating questions, putting together an assessment, then scheduling it, taking it and seeing the results. Start off with some first-hand experience that will give you a firm footing for learning more at the conference.


Jim Parry, Owner and Chief Executive Manager of Compass Consultants, LLC

features-functions-2017Advanced Test Item Writing Workshop: Learn how to test more than just knowledge

Writing test items is difficult, but trying to make them check more than knowledge is a huge challenge.

Join Certified Performance Technologist Jim Parry  — an expert user of Questionmark technologies — in this fast-paced, high-powered , which will present a review of the basics of testing and provide hands-on practice to help you turn low complexity, knowledge-based test items into higher complexity, performance-based items following Bloom’s Taxonomy and Gagne’s Nine Events of Instruction.

Conference Registration Tuition:

  • You can save $200 by registering for the conference on or before January 18. You can sign up for a workshop at the same time or add in a workshop later. It’s up to you! Pre-conference workshop or bootcamp add-ons available during the registration process

Many shades of grey in sensitivity of assessment data

Shades of greyJohn Kleeman HeadshotPosted by John Kleeman

Under data protection law in Europe and increasingly other jurisdictions, “sensitive” personal data has to be given special protection. What does this mean for assessments?

How is sensitive data defined?

The idea behind the concept of “sensitive” or “special” categories of data is that there are some sorts of personal data that if misused could have severe consequences on an individual’s rights or social environment. For instance, information on a living person’s health, racial origin, sexual orientation and political opinions is usually considered sensitive, and special care is needed in processing this information.

At present within Europe, there are minor national differences as to what information is considered sensitive but the forthcoming General Data Protection Regulation (GDPR) should make this more uniform.  In the US, the HIPAA patient privacy law defines the concept of protected health information (PHI). Most PHI would likely also be sensitive under European rules, but HIPAA does not protect political or other non-health information, whereas Europe’s sensitive personal data rules can.

When is assessment data sensitive?

The results of most ordinary skill or knowledge assessments is not sensitive personal data, but here are some ways in which assessment data could or will be sensitive.

  • Health diagnosis. The results of some assessments used in mental health clearly are sensitive. What about psychometric assessments that assess mental state and personality, arguably an aspect of health? This is a grey area, and results from such assessments might be sensitive.
  • Sensitive surveys. If you ask surveys about someone’s health or political views or other sensitive subjects, the assessment results will be sensitive.
  • Demographic data. Do you ask for racial or ethnic origin to accompany assessments, perhaps in order to gather information to prove your assessments are non-discriminatory? If so, that data is likely sensitive.
  • Identity information gathered to prevent cheating. Depending what information you gather to identify someone or check he/she is not cheating, this might be sensitive. For example the GDPR clearly indicates that biometric information should be considered sensitive.

There will not always be a black and white definition – it may well be grey as to whether data is sensitive or not. For example, in some countries, photographs are considered sensitive due to the fact that you can usually identify race from a photo — but in other countries this is only the case for some photos. The GDPR (which becomes law in 2018) says photos they are only sensitive if used to allow unique identification or authentication.

What does it mean for assessment users if data is sensitive?

Here are three suggestions for what to do if you may be processing sensitive data in an assessment.

Person taking a test1. Get explicit participant consent. Although there are some other legal routes, for most assessment use cases, it’s probably wise to get explicit consent from the participant to process sensitive data. For example, include a question at the start of the assessment identifying what you are going to do with the data, and get the participant’s consent.

2. Since there are consequences including fines for misusing data and in general these will be more severe for sensitive data, it would be wise to take strong technical and organizational measures (e.g. encryption) for sensitive data.

3. It’s also wise to ensure that any processors including assessment vendors are knowledgeable about data protection and that you and they have appropriate legal measures in place to cover data protection.

There are some uncertainties around what data is sensitive and how you should deal with it in an assessment context, but I hope this article helps you understand the likely shades of grey to figure out what might be important in your context.

This blog does not give legal advice – please check with your lawyer for rules that apply to your organization and use case.

Secrets to Measuring & Enhancing Learning Results: Webinar

Julie ProfilePosted by Julie Delazyn

Research has shown that assessments play an important role on learning and retention — and the benefits vary before, during and after a learning experience. No matter where learning occurs, the goal remains the same: ensuring people have the knowledge, skills and abilities to perform well.

So, how can you use assessments to measure and enhance learning within your organization?

Check out our newest 30-minute webinar – and register today!

  • The Secrets to Measuring and Enhancing Learning Results
  • Date & Time: Wed, Dec 7  at 4:00 p.m. UK GMT / 11:00 a.m. US EDT

Join us as we discuss the important role assessments play within the learning process and explore the benefits of using them before, during and after learning. We’ll also give you some useful pointers and resources to take away.

Register for the webinar now. We look forward to seeing you at the session!

U.S. Privacy Shield: Data protection and security

Jamie ArmstrongPosted by Jamie Armstrong

Earlier this year I wrote blog post that summarized some important recent data protection and privacy law developments. Today, I wanted to follow up on that posting by looking particularly at the EU-U.S. Privacy Shield (“Privacy Shield”).

The Privacy Shield came into being to fill the void left by the invalidation of the European Commission decision underpinning the US-EU Safe Harbor Agreement (“Safe Harbor”). From August this year, US organizations have been able to certify compliance to the Privacy Shield – the list of those certified organizations can be viewed here. Questionmark Corporation has certified to the Privacy Shield, and you can view our updated privacy policy here. As was the case for Questionmark’s self-certification to Safe Harbor, our compliance with the Privacy Shield principles is just part of Questionmark’s broader strategy to ensure that relevant international data transfers conform to applicable legal requirements.privcy-shield

The Privacy Shield, as well as other mechanisms such as the EU Model Clauses, provides a way for organizations to comply with EU data protection requirements when personal data is transferred to the US from the EU. Remember that whereas the EU Model Clauses may be relied on for transfers of EU personal data to third countries (i.e. those that are not part of the EEA), the scope of the Privacy Shield is limited to personal data transfers to the US.

The European Commission has produced a helpful guide on the Privacy Shield, aimed at EU citizens, with some key improvements as compared with Safe Harbor being:

  • Greater oversight and monitoring by authorities in the US and EU to ensure compliance, for example by the US Department of Commerce, Department of Transport and Federal Trade Commission;
  • A greater number of ways for individuals to make complaints to enforce their rights without cost, including to an Ombudsman within the US Department of State, via an EU Data Protection Authority, an independent recourse mechanism, and binding arbitration;
  • Additional obligations for participant organizations, like ensuring any third-party transferees provide the same level of protection for personal data as is required by the Privacy Shield.

Although the Privacy Shield includes a number of additional protections for individuals and obligations on organizations, some interest groups remain unconvinced that it is meaningfully different to Safe Harbor and legal challenges in the EU have already been made. With this in mind, organizations that have certified to or may certify to the Privacy Shield will have to monitor EU developments and continue to review their data protection and privacy approaches, so that they are satisfied that there are a sufficient number of means available to them to show adequate protection for EU personal data being transferred to the US. Questionmark’s Privacy Shield certification demonstrates to customers our particular commitment to data protection and security in respect of applicable data.

Check back here for future blog posts on data protection and privacy law issues early next year.

Disclaimer: This blog is provided for general information purposes only and does not constitute legal advice. Any views included are personal to me.

Conference Agenda: Next Gen Authoring, Security and SSO

Julie ProfilePosted by Julie Delazyn

We’ve been carefully crafting the Questionmark Conference 2017 agenda and have so many exciting sessions planned for our most important learning event of the year.

  What’s on the agenda? Here are some highlights:product-interest-2017

  • Staying Ahead of Evolving Security Threats  
  • Role-based Security in Questionmark OnDemand: Managing users and roles effectively
  • Taking Your Test Planning to the Next Level: JTAs and Blueprints 
  • Test Security for Grown-Ups: Enhancing exam integrity through proctoring, recording and monitoring
  • Proving workforce capability in a highly-regulated industry: How PG&E utilizes Questionmark to create valid and reliable testing for training programs
  • And so much more!

webcast-icon-2017Make sure to check out the full agenda and register before January 18 to take advantage of our early bird discounts!

We look forward to seeing you in magical Santa Fe, New Mexico March 21 – 24!


Scary thoughts – Data safer in a cupboard or the Cloud?

halloween-postJohn Kleeman HeadshotPosted by John Kleeman

Is it safe to put your data in the Cloud? Or is it safer in a server in a cupboard under your desk, or even in an internal corporate data center? It’s scary how many organizations think a local server is in fact safer than the cloud, when this isn’t always the case.

I was struck recently by a surprising statistic in the 2016 Data Breach Investigations Report by Verizon. This is a very well respected annual report (available here) on security breaches and how to prevent them.

The surprising and scary data relates to the number of data breaches that are caused by publicly announced bugs in software that organizations have not patched. When security vulnerabilities in software like Microsoft Windows or other widely used software is identified, it is given a CVE number and put in a CVE database. CVEs vary with severity. More serious ones might allow an attacker to gain access to a server without permission; others might be a gap in security that could allow an attacker to increase their access or gain access in conjunction with some other exploit. For almost all CVEs, it’s important to patch them to avoid risk.

In the graph below, you can see that during 2015, around 70 CVEs found in 2015 were exploited in 2015, but an amazing number of other vulnerabilities were also exploited – some dating back many years. So a large number of actual breaches are caused by organizations not fixing vulnerabilities that were found and fixed last year or in previous years.

Graph showing count of CVEs exploited in 2015 by CVE publication date showing a large number of vulnerabilities from prior to 2015 still exploited in 2015

Questionmark, like any other reputable cloud vendor, has a well organized process to keep watch for publicly announced security vulnerabilities. We subscribe to all the appropriate information feeds — and when we hear of a vulnerability, we review the risk and if it is critical, we will deploy very quickly; even if that means disrupting our team’s other projects to ensure security is paramount.

But the graph above shows that some organizations don’t update their systems reliably or often enough. Or else they incorrectly deploy software and get caught by old vulnerabilities. The scary thing is that once a vulnerability is disclosed, attackers set up programs to try attacks based on it indiscriminately.  As the Verizon report says, “attackers automate certain weaponized vulnerabilities and spray them across the internet, sometimes yielding incredible success.”

We can draw an analogy between this and money. Some people are worried that it’s not safe to keep money in the bank, and they store their cash in a cupboard or under a mattress. Those people don’t get hurt by bank fraud but are much more vulnerable to fire, theft and other risks of keeping money at home.

Many organizations that use on-premise software have well-staffed, professional IT departments that implement software fixes promptly and quickly. But the graph above shows that many organizations worldwide do not patch their software for vulnerabilities quickly or at all. You could think that those organizations are essentially keeping their data in a cupboard or under the mattress.They might think it’s safe as it’s under their control, but if you don’t have a well-organized process to fix patches reliably and fast, that safety is only an illusion.pumpkin

Is your assessment data in a cupboard? If so, consider putting it in in the Cloud with a system like Questionmark OnDemand!

Wishing Everyone a Safe and Happy Halloween!

Next Page »