Get tips for combatting test fraud

Chloe MendoncaPosted by Chloe Mendonca

There is a lot of research to support the fact that stepping up investment in learning, training and certification is critical to professional success. A projection from the Institute for Public Policy Research states that ‘between 2012 and 2022, over one-third of all jobs will be created in high-skilled occupations’. This growing need for high-skilled jobs is resulting in a rapid increase in professional qualifications and certifications.

Businesses are recognising the need to invest in skills, spending some £49 billion in 2011 alone on training [figures taken from CBI on skills] — and assessments are a big part of this. They have become widely adopted in helping to evaluate the competence, performance and potential of employees and job candidates. In many industries such as healthcare, life sciences and manufacturing, the stakes are high. Life, limb and livelihood are on the line, so delivering such assessments safely and securely is vital.

Sadly, many studies show that the higher the stakes of an assessment, the higher the potential and motivation to commit test fraud. We see many examples of content theft, impersonation and cheating in the news, so what steps can be taken to mitigate security risks?? What impact do emerging trends such as online remote proctoring have on certification programs? How can you use item banking, secure delivery apps and reporting tools to enhance the defensibility of your assessments?

This October, Questionmark will deliver breakfast briefings in two UK cities, providing the answers to these questions. The briefings will include presentations and discussions on the tools and practices that can be used to create and deliver secure high-stakes tests and exams.

These briefings, due to take place in London and Edinburgh, will be ideal for learning, training and compliance professionals who are using or thinking about using assessments. We invite you to find out more and register for one of these events:

 

Get trustable results: How many test or exam retakes should you allow?

John Kleeman HeadshotPosted by John Kleeman

How many times is it fair and proper for a participant to retake an assessment if they fail?

One of our customers asked me about this recently in regard to a certification exam. I did some research and thought I’d share it  here.

For a few kinds of assessments, you would normally only allow a single attempt, typically if you are measuring something at a specific point in time. A pre-course or post-course test might only be useful if it is taken right before or right after a training course.

Regarding assessments that just give retrieval practice or reinforce learning, you needn’t be concerned. It may be fine to allow as many retakes as people want. The more times they practice answering the questions, the more they will retain the learning.

But how can you decide how many attempts to allow at a certification assessment measuring competence and mastery?

Consider test security

Retakes can jeopardize test security. Someone might take and retake a test to harvest the items to share with others. The more retakes allowed, the more this risk increases.

International Test Commission draft security guidelines say:

“Retake policies should be developed to reduce the opportunities for item harvesting and other forms of test fraud. For example, a test taker should not be allowed to retake a test that he or she “passed” or retake a test until a set amount of time has passed.”

Consider measurement error

All assessment scores have measurement error. A certification exam classifies people as having mastery (pass) or not (failing), but it doesn’t do so perfectly.

If you allow repeat retakes, you increase the risk of classifying someone as a master who is not competent, but  you also decrease the risk of classifying a competent person as having failed. This is because someone can suffer test anxiety or be ill or make a stupid mistake and fail the test despite being competent.

Require participants to wait for retakes

It’s usual to require a time period to elapse before a retake. This  stops people from  using quick, repeated retakes to take unfair advantage of measurement error. It also encourage reflection and re-learning before the next attempt. Standard 13.6 in the Standards for Educational and Psychological Testing says:

“students. . . should have a reasonable number of opportunities to succeed. . . the time intervals between the opportunities should allow for students to have the opportunity to obtain the relevant instructional experiences.”

If we had a perfectly reliable assessment, there would be no concern about multiple attempts. Picking the number of attempts is a compromise between what is fair to the participants and the limitations of our resources as assessment developers.

Think about test preparation

Could your retake policy affect how people prepare for the exam?

If retakes are easily available, some participants might prepare less effectively, hoping that they can “wing it” since  they can retake at will.  On the other hand, if retakes are limited, this could increase test anxiety and stress. It could also increase the motivation to cheat.

What about fairness?

Some people suffer test anxiety, some people make silly mistakes on the test or use poor time management, and some may be not at their full capacity on the day of the exam. It’s usually fair to offer a retake in such situations. If you do not offer sufficient opportunities to retake, this will impact the face validity of the assessment: people might not consider it fair.

If your exam is open the public, you may not be able to limit retakes. Imagine a country where you were not allowed to retake your driving test once you’d failed it 3 times! It might make the roads safer, but most people wouldn’t see it as equitable.

In my next post on this subject, I will share what some organizations do in practice and offer some steps for arriving at an answer that will be suitable for your organization.

PwC’s half-time strategy for ensuring effective diagnostic tests

John Kleeman HeadshotPosted by John Kleeman

I shared in a previous blog article how PwC explained at the recent Questionmark Users Conference some good practices for diagnostic tests that allow people to test out of training. In this article, I’ll explain what they call their “half-time” strategy, which enables even greater time savings.

A downside of a test-out strategy is that everyone, including the people who fail, have to spend time taking the test. PwC have found that in some cases, particularly if the test is difficult to pass, having a test-out can actually increase the total time taken for pre-testing and training. Although the test may add learning value in itself, the aim of testing is to reduce time spent, not to increase it!

PwC’s “half time strategy” deals with this issue by dividing the test into two halves; if a test taker does poorly in the first half, they don’t attempt the second half as they would have no chance of passing the test. This reduces the time spent in taking the test for many people, and so improves the overall efficiency of the process.

See the diagram below for a visual representation.

PwC chart 2

Essentially, if someone is doing badly at the test, it’s best to tell them so and stop wasting their time. If the test taker is doing poorly on the first half, cut their losses right then and there.

Suppose, for example, the pass score is 80% and you need to get 14 out of 17 questions right to be able to pass the test and test out of the training. Then you might divide the test into a first half with 9 questions and a second half with 8 questions. If you don’t score at least 6 out of 9 in the first half, then even if you get all the questions in the second half right, you couldn’t pass it. So people who score less than 6 in the first half of the test needn’t take the second half.

In my previous post, I gave an example of 1,000 people who need to take compliance training and could test out by taking a 20-minute test. It takes 333 hours for people to complete the test, but in the example I gave, time was saved overall by reducing time spent training.

Using the half-time strategy, suppose that 45% of those people drop out after the first half of the test — so they only need to spend 10 minutes on the test rather than 20 minutes. Then you save an extra 75 hours of test taker time in aggregate (450 x 10 minutes). This approach also prevents learner frustration by not requiring someone to continue with a test they are going to fail.

The half-time strategy is very easy to implement in Questionmark and seems a useful technique for minimizing the total amount of time spent on diagnostic testing and subsequent training.

I hope this idea is useful for you.

You can see some other articles inspired by PwC’s practices on the Questionmark blog at Applying the principles of item and test analysis to yield better results and How many items are needed for each topic in an assessment? How PwC decides.

Conference Close-up: Questionmark Boot Camp for Beginners

Posted by Joan Phaup

For people attending the Questionmark Users Conference who are new to our software, we’re pleased to be offering a one-day pre-conference workshop that will get them off to a great start!

Our own Rick Ault, who has led many a Questionmark product training course, will lead Questionmark Boot Camp: Basic Training for Beginners at the Ritz Carlton New Orleans on Tuesday, March 20. He shared some details about this with me a few days ago:

“Boot Camp” sounds a little scary! What will happen during that workshop?

We call it boot camp in the same sense that a military boot camp gives you basic training in order to get more specialized skills. Our boot camp will give new Questionmark users the basic skills and knowledge so that they can get as much as possible out of the conference. We’ll give people a  solid foundation, so that they can seek more specialized information and drill a little deeper during the conference into the areas that interest them most. We want to empower attendees to make the most of their conference.

Rick Ault

Rick Ault

How much do you expect people  to learn in one day?

They’ll get a broad overview of how things work and pick up some skills by actually doing things and getting some practice. We’ll cover basic concepts such as how you create a question, how you group questions into an assessments, then publish it, schedule it, deliver it and get the results back. From start to finish, they’ll bulid up a foundation that will help them have meaningful discussions during the conference. We’re distilling our regular three-day training course into one day, so it’s not nearly as deep, but the scope of the class is the same as the longer course.

What do you hope people will take away from their day in class?

I hope they’ll take away a good understanding of how Questionmark works. I hope they’ll be able to go back to work being able to create some basic assessments, and that they’ll be on their way to doing more advanced things if they decide to go that route.

How should participants prepare for Boot Camp?

Get a good night’s rest, bring your computer and make sure it works!

What are you looking forward to at the conference?

It’s always nice to shepherd my flock – to see my past training attendees and see how they’ve grown and how they’re using Perception today…They teach me new things sometimes!

If you have not signed up for the conference yet, we hope you will sign up in the coming week to save $100 on your registration fee! (Early-bird registration ends next Friday, January 27.)