Six tips to increase content validity in competence tests and exams

Posted by John Kleeman

Content validity is one of the most important criteria on which to judge a test, exam or quiz. This blog post explains what content validity is, why it matters and how to increase it when using competence tests and exams within regulatory compliance and other work settings.

What is content validity?

An assessment has content validity if the content of the assessment matches what is being measured, i.e. it reflects the knowledge/skills required to do a job or demonstrate that the participant grasps course content sufficiently.

Content validity is often measured by having a group of subject matter experts (SMEs) verify that the test measures what it is supposed to measure.

Why does content validity matter?

If an assessment doesn’t have content validity, then the test isn’t actually testing what it seeks to, or it misses important aspects of job skills.

Would you want to fly in a plane, where the pilot knows how to take off but not land? Obviously not! Assessments for airline pilots take account all job functions including landing in emergency scenarios.

Similarly, if you are testing your employees to ensure competence for regulatory compliance purposes, or before you let them sell your products, you need to ensure the tests have content validity – that is to say they cover the job skills required.

Additionally to these common sense reasons, if you use an assessment without content validity to make decisions about people, you could face a lawsuit. See this blog post which describes a US lawsuit where a court ruled that because a policing test didn’t match the job skills, it couldn’t be used fairly for promotion purposes.

How can you increase content validity?

Here are some tips to get you started. For a deeper dive, Questionmark has several white papers that will help, and I also recommend Shrock & Coscarelli’s excellent book “Criterion-Referenced Test Development”.

  1. Conduct a job task analysis (JTA). A JTA is a survey which asks experts in the job role what tasks are important and how often they are done. A JTA gives you the information to define assessment topics in terms of what the job needs. Questionmark has a JTA question type which makes it easy to deliver and report on JTAs.
  2. Define the topics in the test before authoring. Use an item bank to store questions, and define the topics carefully before you start writing the questions. See Know what your questions are about before you deliver the test for some more reasoning on this.
  3. You can poll subject matter experts to check content validity for an existing test. If you have an existing assessment, and you need to check its content validity, get a panel of SMEs (experts) to rate each question as to whether it is  “essential,” “useful, but not essential,” or “not necessary” to the performance of what is being measured. The more SMEs who agree that items are essential, the higher the content validity. See Understanding Assessment Validity- Content Validity for a way to do this within Questionmark software.
  4. Use item analysis reporting. Item analysis reports flag questions which are don’t correlate well with the rest of the assessment. Questionmark has an easy to understand item analysis report which will flag potential questions for review. One of the reasons a question might get flagged is because participants who do well on other questions don’t do well on this question – this could indicate the question lacks content validity.
  5. Involve Subject Matter Experts (SMEs). It might sound obvious, but the more you involve SMEs in your assessment development, the more content validity you are likely to get. Use an assessment management system which is easy for busy SMEs to use, and involve SMEs in writing and reviewing questions.
  6. Review and update tests frequently. Skills required for jobs change quickly with changing technology and changing regulations.  Many workplace tests that were valid two years ago, are not valid today. Use an item bank with a search facility to manage your questions, and review and update or retire questions that are no longer relevant.

I hope this blog post reminds you why content validity matters and gives helpful tips to improve the content validity of your tests. If you are using a Learning Management System to create and deliver assessments, you may struggle to obtain and demonstrate content validity. If you want to see how Questionmark software can help manage your assessments, request a personalized demo today.

 

GDPR is coming. Are you ready?

Posted by Julie Delazyn

Don’t get left behind as the most important change in data privacy takes effect May 2018. The new General Data Protection Regulation (GDPR) intends to strengthen and unify privacy and data protection and any organization that stores or manages data about Europeans will need to comply.

With eye-watering regulatory fines of up to €20 million or 4% of global annual turnover (whichever is greater), a credible compliance strategy is essential.

Join us for a FREE 45 minute Webinar July 26, 2017, to understand how online assessments can help you meet your GDPR challenges.

The webinar will cover:

  • What the GDPR is and who it impacts
  • Why you should care about GDPR compliance
  • How to overcome the challenges presented by GDPR — including the learning curve for your employees
  • How assessment can help mitigate GDPR risks and aid your compliance strategy
  • Considerations for implementing assessment management software to aid in compliance

We look forward to speaking to you at the webinar!

6 Tips for trustworthy compliance assessments

Posted by Chloe Mendonca

If you’re responsible for the development or management of compliance tests you have a heavy responsibility on your shoulders. It’s up to you to ensure your tests are both valid and reliable. We’ve spoken about reliability and validity many times here on the Questionmark blog and these really are two of the keys to ensuring your assessment results can be trusted. If your tests don’t measure what they’re designed to or the content doesn’t reflect the required job knowledge, how can you make defensible decisions on the basis of the results?

This infographic shares 6 tips that you should consider implementing if you haven’t already that will help you to develop trustworthy compliance assessments.

 

Click here to get a hi-res copy of this infographic.

To learn more about developing trustworthy assessments, check out the 26-page Questionmark White Paper “Assessment Results You Can Trust”.

Can you be GDPR compliant without testing your employees?

Posted by John Kleeman

The GDPR is a new extra-territorial, data protection law which imposes obligations on anyone who processes personal data on European residents. It impacts companies with employees in Europe, awarding bodies and test publishers who test candidates in Europe, universities and colleges with students in Europe and many others. Many North American and other non-European organizations will need to comply.

See my earlier post How to use assessments for GDPR compliance for an introduction to GDPR. The question this blog post addresses is whether it’s practical for a large organization to be compliant with the GDPR without giving tests and assessments to their employees?

I’d argue that for most organizations with 100s or 1000s of employees, you will need to test your employees on your policies and procedures for data protection and the GDPR. Putting it simply, if you don’t and your people make mistakes, fines are likely to be higher.

Here are four things the GDPR law says (I’ve paraphrased the language and linked to the full text for those interested):


1. Organizations must take steps to ensure that everyone who works for them only processes personal data based on proper instructions. (Article 32.4)

2. Organizations must conduct awareness-raising and training of staff who process personal data (Article 39.1). This is extended to include “monitoring training” for some organizations in Article 47.2.

3. Organizations must put in place risk-based security measures to ensure confidentiality and integrity and must regularly test, assess and evaluate the effectiveness of these measures. (Article 32.1)

4. If you don’t follow the rules, you could be fined up to 20 million Euros or 4% of turnover. How well you’ve implemented the measures in article 32 (i.e. including those above) will impact how big these fines might be. (Article 83.2d)


So let’s join up the dots.

Firstly, a large company has to ensure that everyone who works for it only processes data based on proper instructions. Since the nature of personal data, processing and instructions each have particular meanings, this needs training to help people understand. You could just train and not test, but given that the concepts are not simple, it would seem sensible to test or otherwise check their understanding.

A company is required to train its employees under Article 39. But the requirement in Article 32 is for most companies stronger. For most large organizations the risk of employees making mistakes and the risk of insider threat to confidentiality and integrity is considerable. So you have to put in place training and other security measures to reduce this risk. Given that you have to regularly assess and evaluate the effectiveness of these measures, it seems hard to envisage an efficient way of doing this without testing your personnel. Delivering regular online tests or quizzes to your employees is the obvious way to check that training has been effective and your people know, understand and can apply your processes and procedures.

Lastly, imagine your company makes a mistake and one of your employees causes a breach of personal data or commits another infraction under the GDPR? How are you going to show that you took all the steps you could to minimize the risk? An obvious question is whether you did your best to train that employee in good practice and in your processes and procedures? If you didn’t train, it’s hard to argue that you took the proper steps to be compliant. But even if you trained, a regulator will ask you how you are evaluating the effectiveness of your training. As a regulator in another context has stated:

“”where staff understanding has not been tested, it is hard for firms to judge how well the relevant training has been absorbed”

So yes, you can imagine a way in which a large company might manage to be compliant with the GDPR without testing employees. There are other ways of checking understanding, for example 1:1 interviews, but they are very time consuming and hard to roll out in time for May 2018. Or you may be lucky and have personnel who don’t make mistakes! But for most of us, testing our employees on knowledge of our processes and procedures under the GDPR will be wise.

Questionmark OnDemand is a trustable, easy to use and easy to deploy system for creating and delivering compliance tests and assessments to your personnel. For more information on using assessments to help ensure GDPR compliance visit this page of our website or register for our upcoming webinar on 29 June.

Top 5 benefits of permissions

Bart Hendrickx SmallPosted by Bart Hendrickx

We went over role-based access control and its advantages in some of my earlier blog posts:

As I mentioned previously, roles are made up of permissions. Today, I wanted to share 5 of the top benefits permissions have, in particular, Questionmark OnDemand permissions.

1. Keep your authoring content organized

You can use permissions to define where authors are allowed to create content, such as items (questions) and assessments. That enables you to set up a structure for topic folders and assessment folders, and make sure that authors won’t go outside the folders you gave them access to. That in turn helps you to keep your content organized, which benefits your assessment program’s efficiency.

2. Improve your authors’ user experience

Let’s say that all you want your authors to create are multiple choice questions. You can use a permission to that effect. As a consequence, authors will not be able to see other question types, which makes it less confusing to them. Think: I can only see what I can use. That improves their user experience.

3. Improve your reporters’ user experience

Similarly, for several reports, you can define which reports your reporters can run. By doing that, you guide your reporters to the appropriate reports and avoid confusing them with reports they should not be running.

4. Reduce error and fraud

By using permissions judiciously, you can separate duties in your organization. For example, there are permissions to create users, permissions to create assessments and permissions to schedule assessments. You can use those permissions to define that a user who can create other users cannot create assessments and vice versa. Likewise, the user who can schedule assessments cannot create users. That way, you ensure that each user is responsible for a specific part of the process. When the focus is defined, error is reduced. And if multiple users manage the process together, no-one has full power over everything, which makes fraud less likely.

5. Free up time

Finally, you can use permissions to delegate some of your role responsibilities to others. If you are the main admin user of your Questionmark OnDemand environment, you may want to give a colleague the permission to assign the reporter role to users, so that you do not have to do it repeatedly. However, you may not want to give your colleague the permission to edit the exact permissions that reporters have. By delegating the role assignment to your colleague, you remain in control over what a role can do and you get help with managing who can report. That frees up your time.

 

These are my top 5 reasons. If you can think of other benefits, contribute to this list by leaving a comment below, we’d love to see what benefits you’re experiencing.

How to use assessments for GDPR compliance

Posted by John Kleeman

In about a year’s time, on 25 May 2018, a major new European law, the GDPR, comes into effect. It will update current laws and set new requirements for handling personal data. The GDPR will replace country-specific current data protection laws and will impact any organization anywhere in the world that stores or manages data about people who live or work in Europe. With implementation of the GDPR just a year away, organisations need to prepare themselves now to implement it.

This post explains how an online assessment management system like Questionmark can help your organization comply with the GDPR. We also invite you to a free webinar where you can learn more about how assessments can help you comply.

What will be the fines for failing to comply with the GDPR?

European regulators are suggesting that those who don’t comply will face large fines. The law itself says that fines can be up to €20 million or 4 percent of turnover (whichever is more). The Irish data commissioner has said that there will be no leeway or grace period, and that noncompliant firms will be heavily fined — this under a data protection regime generally considered to be more business-friendly than those in some other countries. Some European regulators are also preparing the ground by issuing high fines under existing data protection regulations. For example, the Italian Data Protection Authority has just issued an €11 million fine.

What is the learning challenge for GDPR?

The GDPR has specific requirements for training personnel and for monitoring the effectiveness of the training. In addition, most companies will be incentivized to train their employees to ensure that personnel understand the GDPR and their own company policies/procedures – since failing to follow the rules will make companies liable for fines.

The GDPR legislation is 88 pages long and the rules are complex. Within most companies, many different departments will be responsible for handling and processing data: HR, IT, sales and marketing, technology and product development and more. All these departments will need to climb the learning curve to know and understand how the GDPR applies to them.

Thus, a key part of the compliance challenge for GDPR will be ensuring that your personnel learn and understand the rules.

How assessments can help

An assessment management system like Questionmark’s lets you:

  • Create quizzes, tests and exams on the GDPR’s or your company’s policies and procedures
  • Deliver them to personnel at home, work or on mobile devices
  • Analyze the results to help you find weaknesses or potential problems
  • Store the results as trustable evidence that your people understand

Trustable, online assessments can help in many ways with GDPR compliance.  They are the best way to help ensure your employees understand the rules and to reduce the likelihood of mistakes that lead to fines. Assessments are also the best way of providing evidence that you did train your personnel well. This evidence could mitigate the amount of a fine, in the event of a privacy misstep.

Questionmark’s webinar

To help you understand more about how online assessments can help you meet your GDPR challenge, we invite you to a webinar presented by myself and Questionmark’s UK Marketing Manager, Chloe Mendonca on Thursday 29th June, 2017.

The webinar will cover:

  • What the GDPR is and who it impacts
  • Why you should care about GDPR compliance
  • How to overcome the challenges presented by GDPR — including the learning curve for your employees
  • How assessment can help mitigate GDPR risks and aid your compliance strategy
  • Considerations for implementing assessment management software to aid in compliance

We look forward to speaking to you at the webinar. Click here to register.

GDPR Webinar Image

 

« Previous PageNext Page »