Seven Ways Assessments Fortify Compliance

Posted by John Kleeman
Picture of a tablet being used to take an assessment with currency symbols adjacentWhy do most of the world’s banks, pharmaceutical companies, utilities and other large companies use online assessments to test the competence of their employees?

It’s primarily because compliance fines round the world are high and assessments reduce the risk of regulatory compliance failures. Assessments also give protection to the organization in the event of an individual mis-step by proving that the organization had checked the individual’s knowledge of the rules prior to the mistake.

Here are seven reasons companies use assessments from my experience:

1. Regulators encourage assessments 

Some regulators require companies to test their workforce regularly. For example the US FDIC says in its compliance manual:

“Once personnel have been trained on a particular subject, a compliance officer should periodically assess employees on their knowledge and comprehension of the subject matter”

And the European Securities and Market Authority says in its guidelines for assessment of knowledge and competence:

“ongoing assessment will contain updated material and will test staff on their knowledge of, for example, regulatory changes, new products and services available on the market”

Other regulators focus more on companies ensuring that their workforce is competent, rather than specifying how companies ensure it, but most welcome clear evidence that personnel have been trained and have shown understanding of the training.

People sitting at desks with computers taking tests2. Assessments demonstrate commitment to your workforce and to regulators

Many compliance errors happen because managers pay lip service to following the rules but indicate in their behavior they don’t mean it. If you assess all employees and managers regularly, and require additional training or sanctions for failing tests, it sends a clear message to your workforce that knowledge and observance of the rules is genuinely required.

Some regulators also take commitment to compliance by the organization into account when setting the level of fines, and may reduce fines if there is serious evidence of compliance activities, which assessments can be a useful part of. For example the German Federal Court recently ruled that fines should be less if there is evidence of effective compliance management.

3. Assessments find problems early

Online assessments are one of the few ways in which a compliance team can touch all employees in an organization. You can see results by team, department, location or individual and identify who understands what and focus in on weak areas to look at improving. There is no better way to reach all employees.

4. Assessments document understanding after training

Many regulators require training to be documented. Giving someone an assessment after training doesn’t just confirm he or she attended the course but confirms they understood the training.

5. Assessments increase retention of knowledge and reduce forgetting

Can you remember everything you learned? Of course, none of us can!

There is good evidence that quizzes and tests increase retention and reduce forgetting. This is partly because people study for tests and so remind themselves of the knowledge they learned, which helps retain it. And it is partly because retrieving information in a quiz or test makes it easier to retrieve the same information in future, and so more likely to be able to apply in practice when needed.

6. By allowing testing out, assessments reduce the time and cost of compliance trainingTake test. If pass, skip training. Otherwise do training.

Many organizations permit employees to “test out” of compliance training. People can take a test and if they demonstrate good enough knowledge, they don’t need to attend the training. This concentrates training resources and employee time on areas that are needed, and avoids demoralizing employees with boring compliance training repeating what they already know.

7. Assessments reduce human error which reduces the likelihood of a compliance mis-step

Many compliance failures arise from human error. Root cause analysis of human error suggests that a good proportion of errors are caused by people not understanding training, training being missing or people not following procedures. Assessments can pick up and prevent mistakes caused by people not understanding what they should do or how to follow procedures, and so reduce the risk of error.

 

If you are interested in learning more about the reasons online assessments mitigate compliance risk, Questionmark are giving a webinar “Seven Ways Assessments Fortify Compliance” on April 11th. To register for this or our other free webinars, go to www.questionmark.com/questionmark_webinars.

Six tips to increase content validity in competence tests and exams

Posted by John Kleeman

Content validity is one of the most important criteria on which to judge a test, exam or quiz. This blog post explains what content validity is, why it matters and how to increase it when using competence tests and exams within regulatory compliance and other work settings.

What is content validity?

An assessment has content validity if the content of the assessment matches what is being measured, i.e. it reflects the knowledge/skills required to do a job or demonstrate that the participant grasps course content sufficiently.

Content validity is often measured by having a group of subject matter experts (SMEs) verify that the test measures what it is supposed to measure.

Why does content validity matter?

If an assessment doesn’t have content validity, then the test isn’t actually testing what it seeks to, or it misses important aspects of job skills.

Would you want to fly in a plane, where the pilot knows how to take off but not land? Obviously not! Assessments for airline pilots take account all job functions including landing in emergency scenarios.

Similarly, if you are testing your employees to ensure competence for regulatory compliance purposes, or before you let them sell your products, you need to ensure the tests have content validity – that is to say they cover the job skills required.

Additionally to these common sense reasons, if you use an assessment without content validity to make decisions about people, you could face a lawsuit. See this blog post which describes a US lawsuit where a court ruled that because a policing test didn’t match the job skills, it couldn’t be used fairly for promotion purposes.

How can you increase content validity?

Here are some tips to get you started. For a deeper dive, Questionmark has several white papers that will help, and I also recommend Shrock & Coscarelli’s excellent book “Criterion-Referenced Test Development”.

  1. Conduct a job task analysis (JTA). A JTA is a survey which asks experts in the job role what tasks are important and how often they are done. A JTA gives you the information to define assessment topics in terms of what the job needs. Questionmark has a JTA question type which makes it easy to deliver and report on JTAs.
  2. Define the topics in the test before authoring. Use an item bank to store questions, and define the topics carefully before you start writing the questions. See Know what your questions are about before you deliver the test for some more reasoning on this.
  3. You can poll subject matter experts to check content validity for an existing test. If you have an existing assessment, and you need to check its content validity, get a panel of SMEs (experts) to rate each question as to whether it is  “essential,” “useful, but not essential,” or “not necessary” to the performance of what is being measured. The more SMEs who agree that items are essential, the higher the content validity. See Understanding Assessment Validity- Content Validity for a way to do this within Questionmark software.
  4. Use item analysis reporting. Item analysis reports flag questions which are don’t correlate well with the rest of the assessment. Questionmark has an easy to understand item analysis report which will flag potential questions for review. One of the reasons a question might get flagged is because participants who do well on other questions don’t do well on this question – this could indicate the question lacks content validity.
  5. Involve Subject Matter Experts (SMEs). It might sound obvious, but the more you involve SMEs in your assessment development, the more content validity you are likely to get. Use an assessment management system which is easy for busy SMEs to use, and involve SMEs in writing and reviewing questions.
  6. Review and update tests frequently. Skills required for jobs change quickly with changing technology and changing regulations.  Many workplace tests that were valid two years ago, are not valid today. Use an item bank with a search facility to manage your questions, and review and update or retire questions that are no longer relevant.

I hope this blog post reminds you why content validity matters and gives helpful tips to improve the content validity of your tests. If you are using a Learning Management System to create and deliver assessments, you may struggle to obtain and demonstrate content validity. If you want to see how Questionmark software can help manage your assessments, request a personalized demo today.

 

The Ultimate Guide To Using Assessments for Compliance [eBook]

ebookJulie ProfilePosted by Julie Delazyn

With increasing regulatory requirements, compliance is becoming more and more of a priority for many organizations.

Without regular testing, how do you know what your employees know? And in the case of an audit or an emergency, is it good enough to have had the participant sign off saying that they’ve attended training and understand the content? Most organizations today see online assessments as a critical part of their compliance programs.

Download your complimentary copy of the eBook: Using Assessments for Regulatory Compliance to learn about the most useful applications of assessments in a compliance program and best practice recommendations for using them.

SAP to resell Questionmark software to its customers

John Kleeman HeadshotPosted by John Kleeman

Yesterday we were very happy to  announce the signing of a global reseller agreement with SAP AG, under which SAP will resell our technologies to its customers. You can see the press release here.

Under the agreement, SAP will resell assessment technologies from Questionmark under the name SAP® Assessment Management by Questionmark.
SAP® Assessment Management by Questionmark will enable SAP customers to create, deliver and analyze surveys, quizzes, tests and exams. The software  will complement SAP Learning Solution and SuccessFactors Learning, allowing use of assessments for purposes such as certification, regulatory compliance, and health and safety training for a company’s internal tracking purposes.

SAP will deliver the application on-premise and as a cloud offering to give customers a secure, collaborative environment for creating learning assessments in multiple languages and delivering to a wide variety of browsers and devices. We then anticipate that SAP® Assessment Management by Questionmark users will be able to evaluate results and inform stakeholders with timely, meaningful analyses and reports on items and tests.

This agreement builds on our established relationship as an SAP software solution and technology partner, as well as the longstanding compatibility of our assessment management software with both SAP Learning Solution and SuccessFactors Learning.

I am delighted that this agreement will help SAP customers get even more value from their investments in enterprise learning software. I’ve been working closely with SAP as they have reviewed our technology and I admire SAP’s commitment to quality in learning and technology and the strength and quality of their team.

I’m pleased to share a quote within the press release from Markus Schwarz, SVP and global head of SAP Education:

“Since the need to assess learning is at an all-time high, the addition of this offering to our portfolio of collaborative learning software from SAP Education is well-timed indeed. And because it works together with SuccessFactors Learning and SAP Learning Solution, the new SAP Assessment Management application by Questionmark will dovetail perfectly with our strategy to bring cloud-based learning to SAP customers and partners worldwide.”

To Your Health: Are regulators pleased if everyone passes a test?

John Kleeman Headshot

Posted by John Kleeman

We have all read about the huge regulator fines imposed  in the financial services industry, but regulatory activity in the health care industry is also rising rapidly. It seems to me that in this and other industries, there is a sea change: regulators are holding companies more to account and checking much more carefully that they are following the rules.

A stunning example of this is shown in the graph below from the US FDA showing the rise in the formal warning letters it has issued across industries in recent years – from 471 in fiscal year 2007 to 4,882 in fiscal year 2012.

FDA Warning Letters rising fiscal years 2007-2012

I have not read all these thousands of letters (!), but here are a few interesting excerpts relating to failing to train and assess well enough. You can see all warning letters on the FDA website:

From a letter to a medical devices company in 2012

“Failure to document that all personnel are trained to adequately perform their assigned responsibilities … Specifically, no documentation was provided which demonstrated that manufacturing technicians or customer service personnel received training for their assigned duties.”

From a letter to a contract pharmaceutical laboratory in 2012:

“your laboratory should establish a procedure for requiring, documenting, and periodically assessing all training”

From a letter to a pharmaceutical manufacturer in 2012:

“Your employees were not adequately trained in CGMPs as evidenced by the deficiencies listed in this letter. In your response to this letter, provide a plan to develop an ongoing and robust CGMP training program for your personnel, including an explanation of how you will assess training effectiveness.“

And from an older letter to a US blood bank:

There is no evidence an annual written test has been given to all laboratory personnel as required by the “Training Program” procedure for the production laboratory.

These are interesting evidence that regulators expect assessments or other evidence of training. I was prompted to write this blog article by one other excerpt from regulatory action by the FDA. This example fascinated me not because the organization failed to assess, but because it did assess, and the assessments failed to pick up the problems.

In a letter to a blood services organization in 2012, the FDA said (red highlighting by me):

During an inspection …  FDA discovered that annual competency reviews and/or QA reviews did not detect that employees were not correctly performing all steps of testing blood samples. One test was repeatedly performed incorrectly by many employees beginning 2007, and another test was repeatedly performed incorrectly by many employees since April 2008. FDA’s review of the competency assessments for those employees performing those tests found that none failed the assessment.

So in this case, employees were being tested. But they were all passing, in spite of not doing their job correctly. This highlights a key requirement when using assessments:  well designed assessments are the best way of measuring competence and understanding. But you need to make sure that your assessments are well designed, valid and reliable. A poor quality assessment can give false comfort.

A sensible regulator will not only want to see that you assess your employees’ competence, but also that you use professional techniques to create, deliver and report on those assessments to ensure that they match the requirements for the job.  It is of course desirable that all your employees pass competency assessments, but only if those assessments truly measure the key requirements for the job.

For more background on good practice using assessments in the health care industry, read the previous To Your Health posts listed below. You can also read Questionmark’s best practice white papers for advice on providing assessments that get effective results.

Webinars on Mobile Assessment, Regulatory Compliance and Browser-Based Authoring

Chloe MendoncaPosted by Chloe Mendonca

Our current UK Web seminars offer technology updates and some guidance on how to use assessment to benefit your organisation.

All of these one-hour webinars are scheduled for 11 a.m. London BST


What’s New in Questionmark Live Browser-Based Authoring? – June 12 and July 4
Subject matter experts (SMEs) can easily write questions and complete assessments anytime, anywhere thanks to our easy-to-use, browser-based authoring tool. Join this session to learn about the newest authoring features in Questionmark Live.

Creating Assessments for Mobile Delivery — Wednesday, 29 May and Thursday, 20 June
You can deliver a single assessment to many different types of devices, and process the results centrally. Join this webinar to learn about a cost-effective and flexible way to provide assessments for people on the move. You will find out how to create assessments that sit well on a small screen.

Using Assessments to Mitigate Risk and Ensure Regulatory Compliance – Thursday, 6 June and Wednesday 26 June
This session will explain how assessments can help your organization demonstrate commitment to comply with laws, warn of a lack of knowledge before it impacts the business and provide evidence that appropriate precautions such as training have taken place.

Introduction to Questionmark’ s Assessment Management System – Thursday, 30 May and Wednesday, 19 June
This introductory web seminar explains and demonstrates key features and functions available in Questionmark OnDemand and Questionmark Perception. It will show how you can use our assessment management technologies within your organisation.

Click here to sign up for any of these webinars.