Save Time by Testing Out of Compliance Training?

Posted by John Kleeman

There is nothing worse for an employee than attending a compliance training session about material they already know. (Or at least sometimes it feels that way!)

If employees already know something well, then training them in it again is a waste of resources and motivation. By forcing people to attend training they feel is unnecessary, you reduce the credibility of your whole compliance initiative. People feel that you are just crossing a task off your list, not really caring about what matters to them and the business.

Diagnostic tests can provide a way of identifying what employees know or don’t know, and so allows employees to “test out” of training that they do not need to take. So for example, if a training course covers three topics, then some employees may be able to show– by passing a test — that they already know one or more topics and can skip part or all of the training.

Questionmark Perception screenshot

You’ll need to check your regulations to confirm that this is permissible for you, but most regulators will see sense here. For instance, an American Banker’s Association Bank Compliance magazine article mentions various US financial regulators who regard testing out to be acceptable:

“Is testing out considered an acceptable alternative to compliance training? The OTS said yes, testing out is acceptable. The OCC agreed, provided the testing program is well structured. However, exceptions may exist for required training such as the Bank Secrecy Act. The FDIC also agreed, commenting that a well-planned, -implemented, and -maintained compliance training program will prevent or reduce regulatory violations, will provide cost efficiencies, and is a sound business practice.”

Obviously if you follow this approach, you’ll need to ensure that your tests are appropriate and sufficient – that they measure the right knowledge and skills that the compliance training would otherwise cover.

But a simple calculation suggests that testing out can save a lot of time. The average employee works around 2,000 hours in a year.


· You need to train 10,000 employees

· 20% of them already know the subject well

· Avoiding training will save on average an hour of someone’s time


Conclusion: By allowing people to test out, you will save 2,000 hours of time or a whole year of someone’s time.

I know many organizations already use Questionmark software to allow employees to “test out” from training. (One of our customers presented a case study at the 2011 Questionmark Users Conference showing that they had saved US$40,000 on a project by doing just this.) But if you are not already allowing testing out for compliance training purposes, it could be worth considering.

Back from Brussels: Remembering a great conference

Posted by Jane Townsend

Returning  from European Users Conference in Brussels, I am amazed at how quickly the time has gone by since we arrived there a few days ago.

This being my first conference, I found it really insightful and would like to thank our many guest speakers for sharing their knowledge and time. Without them there would be no conference.  I’d also like to say a big thank you to all the Questionmark staff who were so supportive throughout…Great Job!

We covered so many topics during our two days together— everything from news about Questionmark Analytics and Mobile Apps to a fascinating general session on ways to balance flexibility, participant experience, logistics and security when considering test authoring and delivery options. Getting together with customers gave us the opportunity to take stock of the innovative technologies and techniques that are changing the world of learning and assessment. 

The closing session went down well with staff and clients alike, emphasizing constant improvements to our technologies and services.

Our conferences are never complete without some fun, and we had plenty during our  “Speak Easy” event,  where fortunes were won and lost in seconds. Who will forget being  enveloped by a very large snake or losing ties, wrist watches and even their minds to a skillful pickpocket?  For more photos from the conference, check out our flickr page.

Where’s the evidence for assessment?

Posted by John Kleeman

I’m always on the lookout for hard evidence that assessment improves learning outcomes, and I’m indebted to Denise Whitelock, Lester Gilbert and Veronica Gale for alerting me to some powerful evidence in their research report at the 2011 CAA Conference. This Australian study looked at more than 1500 students taking part in an applied maths course and showed that taking formative quizzes during a course improved learning outcomes.

The study was conducted by two economics lecturers, Dr Simon Angus and Judith Watson, and is titled Does regular online testing enhance student learning in the numerical sciences? Robust evidence from a large data set. It was published in the peer-reviewed British Journal of Educational Technology Vol 40 No 2, 255-272 in 2009.

Angus and Watson introduced a series of 4 online, formative quizzes into the course, and wanted to determine whether students who took the quizzes learned more and did better on the final exam than those who didn’t. The interesting thing about the study is that they used a statistical technique (ordinary least squares regression). This allowed them to estimate the effect of several different factors, and isolate the effects of taking the quizzes from the previous mathematical experience of the students, their gender and their general level of effort to determine which impacted the final exam score most.

You can see a summary of their findings in the graph below, which shows the estimated coefficients for four of the main factors, all of which had a statistical significance of p < 0.01.

Factors associated with final exam score graph

You can see from this graph that the biggest factor associated with final exam success was how well students had done in the midterm exam, i.e. how well they were doing in the course generally. But students who took the 4 online quizzes learned from them and did significantly better. The impact of taking or not taking the quizzes was broadly the same as the impact of their prior maths education: quite reasonable and significant.

We know intuitively that formative quizzes help learning, but it’s nice to see a statistical proof that – to quote the authors – “exposure to a regular (low mark) online quiz instrument has a significant and positive effect on student learning as measured by an end of semester examination”.

Security tips for defensible assessments

Posted by Joan Phaup

Validity and reliability, fairness and cut scores are key concerns when it comes to the legal defensibility of assessments. Security is another important element, since cheating and security breaches can throw the validity and fairness of a test into question.

Our white paper, Defensible Assessments: What You Need to Know, aims to help readers determine what defensibility means, to inform them about standards and best practices and to describe how to ensure and evaluate defensibility.

Here are few quick security tips from the paper, which you are welcome to download with our compliments:

  1. Assign role-based security rights so that access to item banks is limited
  2. Determine the topic and assessment folders that authors may access and define how they are allowed to use them. For instance, should a person be able only to view content or should they also be allowed to edit it?
  3. Ensure that the right people are taking a test by requiring participants to log in with a username and password
  4. Set limits on the number of attempts according to the stakes of the assessment
  5. Establish specific dates and times for taking a test
  6. Organize participants into groups and hierarchical subgroups — each with designated default test centers — for simple and secure assessment scheduling
  7. Maintain up-to-date information about participants to avoid confusion in test scheduling and the processing of test results

Questionmark Live Now Has the Matching Question Type

Posted By Doug Peterson

Questionmark Live browser-based authoring has been updated to include the Matching question type in its already long list of available question types. This video shows you how to create a matching question in Questionmark Live, including several key features like scoring per choice and providing feedback per choice.

Please note: This feature will be available in Questionmark Live starting October 13

Proving compliance – not just attendance

Posted by John Kleeman

Many regulators require you to train employees – in financial services, pharmaceuticals, utilities and in health & safety across all industries. You need to train them and when you are audited or if something goes wrong, you need to document that you did the training. To quote the US regulator OSHA: Documentation can also supply an answer to one of the first questions an accident investigator will ask: “Was the injured employee trained to do the job?”

Is it good enough to get the participant to sign something saying that they’ve attended the training or read the safety manual? An excellent blog series on the SafetyXChange says no:

Some companies ask their workers to sign a form after training sessions acknowledging that they understood the lesson and will put it into practice. Don’t let these forms lull you into a false sense of security. “Most workers will just sign these things without even reading them, let alone making sure that they understood everything you told them,” says a health and safety attorney in New York City. This is especially true if the training and instructions are complicated.

In the safety field, a US Appeals Court law case ruled in 2005 (my underlining):

Merely having an individual sign a form acknowledging his responsibility to read the safety manual is insufficient to insure that the detailed instructions contained therein have actually been communicated.

Two good ways to show that someone not only attending the training but also understood it:

Workplace assessment on ladder use

Give employees a test or quiz at the end of the training to confirm that they understood it. This will also give them practice retrieving information to slow the forgetting curve (see Answering Questions directly helps you learn). And it will allow you to pick out people who didn’t get the learning or weak points in the class.

For more practical skills, you might want to observe people to check they understood the training and can practice it, or in the safety world demonstrate that they can do the job safely. For example the screenshot on the right shows how a supervisor can use an iPad to check and log someone’s skill on using a ladder.

My view is that you want to give these kinds of tests for two reasons. First and most importantly, you want to prevent your employees from falling off ladders or making other mistakes. Second, if something does go awry, you want evidence that you’ve trained people well.

If you’re interested in this area you might check out a recent, busy discussion (registration required) on the LinkedIn Compliance Exchange forum. Paraphrasing some of the views there:

Yes, you should give a quiz as it proves attendance – videoing the training is another option.

Yes, you should give a test and regulators in particular the US FDIC are increasingly demanding this

No. Danger of a test is that you need to take action if scores are bad, which may give you a lot of work. Safer not to ask the questions in case you don’t like the answers.

Yes, you should give a test but it can be a very easy and simple one, to check basic understanding and prove attendance

Yes, you should test, as well as confirming understanding it will also highlight vulnerabilities in the training

What do you think? Use the reply form below and contribute to the dialog.