10 Reasons Why Frequent Testing Makes Sense

Posted by John Kleeman

It matters to society, organizations and individuals that test results are trustable. Tests and exams are used to make important decisions about people and each failure of test security reduces that trustworthiness.

There are several risks to test security, but two important ones are identity fraud and getting help from others. With identity fraud, someone asks a friend to take the test for them or pays a professional cheater to take the test and pretend to be them. With getting help from others, a test-taker subverts the process and gets a friend or expert to help them with the test, feeding them the right answers. In both cases, this makes the individual test result meaningless and detracts from the value and trustworthiness of the whole assessment process.

There are lots of mitigations to these risks – checking identity carefully, having well trained proctors, using forensics or other reports and using technical solutions like secure browsers – and these are very helpful. But testing more frequently can also reduce the risk: let me explain.

Suppose you just need to pass a single exam to get an important career step – certification, qualification or other important job requirement, then the incentive to cheat on that one test is large. But if you have a series of smaller tests over a period, then it’s more hassle for a test taker to conduct identity fraud or to get help from others each time. He or she would have to pay the proxy test taker several times.  And make sure the same person is available in case photos are captured. And for the expert help you also must reach out more often, and evade whatever security there is each time

There are other benefits too; here is a list of ten reasons why more frequent testing makes sense:

  1. More reliable. More frequent testing contributes to more reliable testing. A single large test is vulnerable to measurement error if a test taker is sick or has an off day, whereas that is less likely to impact frequent tests.
  2. More up to date. With technology and society changing rapidly, more frequent tests can make tests more current. For instance, some IT certification providers create “delta” tests measuring understanding of their latest releases and encourage people to take quarterly tests to ensure they remain up to date.
  3. Less test anxiety. Test anxiety can be a big challenge to some test takers (see Ten tips on reducing test anxiety for online test-takers), and more frequent tests means less is at stake for each one, and so may help test takers be less anxious.
  4. More feedback. More frequent tests give feedback to test takers on how well they are performing and allow them to identify training or continuing education to improve.
  5. More data for testing organization. In today’s world of business intelligence and analytics, there is potential for correlations and other valuable insight from the data of people’s performance in a series of tests over time.
  6. Encourages test takers to target retention of learning. We all know of people who cram for an exam and then forget it afterwards. More frequent tests encourage people to plan to learn for the longer term.
  7. Encourages spaced out learning. There is strong evidence that learning at spaced out intervals makes it more likely knowledge and skills will be retained. Periodic tests encourage revision at regular intervals and so make it more likely that learning will be remembered.
  8. Testing effect. There is also evidence that tests themselves give retrieval practice and aid retention and more frequent tests will give more such practice.
  9. More practical. With online assessment software and online proctoring, it’s very practical to test frequently, and no longer necessary to bring test takers to a central testing center for one off large tests.
  10. Harder to cheat. Finally, as described above, more frequent testing makes it harder to use identity fraud or to get help from others, which reduce cheating.

I think we’re seeing a slow paradigm shift from larger testing events that happen at a single point in time to smaller, online testing events happening periodically. What do you think?

Infographic: Online or Test-Centre Proctoring?

Julie ProfilePosted by Julie Delazyn

For many exams, candidates are required to travel to brick-and-mortar test centers where proctors (or invigilators) supervise the process; However, a new way of proctoring certification exams is rapidly gaining traction. Two of the world’s largest software companies, SAP and Microsoft, offer online proctoring for their certification programs, and many other companies are looking to follow suit.

Do you need to understand the key differences and benefits? Here’s an infographic that explains some of the pros and cons of the two approaches.

Proctoring Infographic

For more on online proctoring, check out this informational page and video below:

 

 

Test Security: Not Necessarily a Question of Proctoring Mode

Austin Fossey-42Posted by Austin Fossey

I recently spent time looking for research studies that analyzed the security levels of online and in-person proctoring. Unfortunately, no one seems to have compared these two approaches with a well-designed study. (If someone has done a rigorous study contrasting these two modes of delivery, please let me know! I certainly may have overlooked it in my research.)

I did learn a lot from the sparse literature that was available, and my main takeaway is this: security is related less to proctoring mode than it is to how much effort the test developer puts into administration planning and test design. Investing in solid administration policies, high-quality monitoring technology, and well-trained proctors is what really matters most for both in-person and online proctoring.

With some effort, testing programs with online proctors can likely achieve levels of security and service comparable to the services offered by many test centers. This came into focus for me after attending several recent seminars about online and in-person proctoring through the Association of Test Publishers (ATP) and Performance Testing Council (PTC).

The Standards for Educational and Psychological Testing provide a full list of considerations for organizations running any type of exam, but here are a few key points gleaned from the Standards and from PTC’s webinar (.wmv) to help you plan for online proctoring:

Control of the Environment

Unless a collaborator is onsite to set up and maintain the test environment, all security controls will need to be managed remotely. Here are suggestions for what you would need to do if you were a test program administrator under those circumstances:

  • Work with your online proctors to define the rules for acceptable test environments.
  • Ensure that test environment requirements are realistic for participants while still meeting your standards for security and comparability between administrations.
  • If security needs demand it, have monitoring equipment sent in advance (e.g., multiple cameras for improved monitoring, scanners to authenticate identification).
  • Clearly communicate policies to participants and get confirmation that they understand and can abide by your policies.
  • Plan policies for scenarios that might arise in an environment that is not managed by the test program administrator or proctor. For example, are you legally allowed to video someone who passes by in the background if they have not given their permission to be recorded? If not, have a policy in place stating that the participant is responsible for finding an isolated place to test. Do you or the proctoring company manage the location where the test is being delivered? If not, have a policy for who takes responsibility and absorbs the cost of an unexpected interruption like a fire alarm or power outage.

You should be prepared to document the comparability of administrations. This might include describing potential variations in the remote environment and how they may or may not impact the assessment results and security.

It is also advisable to audit some administrations to make sure that the testing environments comply with your testing program’s security policy. The online proctors’ incident reports should also be recorded in an administration report, just as they would with an in-person proctor.

Test Materials

You also need to make sure that everything needed to administer the test is provided, either physically or virtually.

  • Each participant must have the equipment and resources needed to take the test. If it is not reasonable to expect the participant to handle these tasks, you need to plan for someone else to do so, just as you would at a test center. For example, it might not be reasonable to expect some participant populations to know how to check whether the computer used for testing meets minimum software requirements.
  • If certain hardware (e.g., secured computers, cameras, scanners, microphones) or test materials (e.g., authorized references, scratch paper) are needed for the assessment design, you need to make sure these are available onsite for the participant and make sure they are collected afterwards.

Accommodations

Accommodations may take the form of physical or virtual test materials, but accommodations can also include additional services or some changes in the format of the assessment.

  • Some accommodations (e.g., extra time, large print) can be controlled by the assessment instrument or an online proctor, just as they would in a test center.
  • Other accommodations require special equipment or personnel onsite. Some personnel (e.g., scribes) may be able to provide their services remotely, but accommodations like tactile printouts of figures for the blind must be present onsite.

Extra effort is clearly needed when setting up an online-proctored test. Activities that might have been handled by a testing center (control of the environment, management of test materials, providing accommodations) now need to be remotely coordinated by the test program staff and proctors; however, the payoffs may be worth the extra effort. If comparable administration practices can be achieved, online-proctored assessments may be cheaper than test centers, offer increased access to participants, and lower the risks of collaborative cheating.

For more on online proctoring, check out this informational page and video below

Get trustable results: How many test or exam retakes should you allow?

John Kleeman HeadshotPosted by John Kleeman

How many times is it fair and proper for a participant to retake an assessment if they fail?

One of our customers asked me about this recently in regard to a certification exam. I did some research and thought I’d share it  here.

For a few kinds of assessments, you would normally only allow a single attempt, typically if you are measuring something at a specific point in time. A pre-course or post-course test might only be useful if it is taken right before or right after a training course.

Regarding assessments that just give retrieval practice or reinforce learning, you needn’t be concerned. It may be fine to allow as many retakes as people want. The more times they practice answering the questions, the more they will retain the learning.

But how can you decide how many attempts to allow at a certification assessment measuring competence and mastery?

Consider test security

Retakes can jeopardize test security. Someone might take and retake a test to harvest the items to share with others. The more retakes allowed, the more this risk increases.

International Test Commission draft security guidelines say:

“Retake policies should be developed to reduce the opportunities for item harvesting and other forms of test fraud. For example, a test taker should not be allowed to retake a test that he or she “passed” or retake a test until a set amount of time has passed.”

Consider measurement error

All assessment scores have measurement error. A certification exam classifies people as having mastery (pass) or not (failing), but it doesn’t do so perfectly.

If you allow repeat retakes, you increase the risk of classifying someone as a master who is not competent, but  you also decrease the risk of classifying a competent person as having failed. This is because someone can suffer test anxiety or be ill or make a stupid mistake and fail the test despite being competent.

Require participants to wait for retakes

It’s usual to require a time period to elapse before a retake. This  stops people from  using quick, repeated retakes to take unfair advantage of measurement error. It also encourage reflection and re-learning before the next attempt. Standard 13.6 in the Standards for Educational and Psychological Testing says:

“students. . . should have a reasonable number of opportunities to succeed. . . the time intervals between the opportunities should allow for students to have the opportunity to obtain the relevant instructional experiences.”

If we had a perfectly reliable assessment, there would be no concern about multiple attempts. Picking the number of attempts is a compromise between what is fair to the participants and the limitations of our resources as assessment developers.

Think about test preparation

Could your retake policy affect how people prepare for the exam?

If retakes are easily available, some participants might prepare less effectively, hoping that they can “wing it” since  they can retake at will.  On the other hand, if retakes are limited, this could increase test anxiety and stress. It could also increase the motivation to cheat.

What about fairness?

Some people suffer test anxiety, some people make silly mistakes on the test or use poor time management, and some may be not at their full capacity on the day of the exam. It’s usually fair to offer a retake in such situations. If you do not offer sufficient opportunities to retake, this will impact the face validity of the assessment: people might not consider it fair.

If your exam is open the public, you may not be able to limit retakes. Imagine a country where you were not allowed to retake your driving test once you’d failed it 3 times! It might make the roads safer, but most people wouldn’t see it as equitable.

In my next post on this subject, I will share what some organizations do in practice and offer some steps for arriving at an answer that will be suitable for your organization.

9 Tips to Prevent Cheating and Ensure Test Security

Chloe MendoncaPosted by Chloe Mendonca

The security of test results is crucial to the validity of test scores. Check out 9 tips to prevent cheating and ensure test security in the  infographic below.

If you’d like more details about these and other tips on ensuring the security and defensibility of your assessments you can download our white paper: Delivering Assessments Safely and Securely. [Free after registration]

Prevent Cheating and Ensure Test Security from Questionmark‘s Slideshare page

Five tips for enhancing test security using technology

Headshot JuliePosted by Julie Delazyn

Test security is a topic that comes up time and time again on education and company forums.  You can improve test security by changing the physical test-taking environment, but you can also use technology to tackle certain security issues.

Here are are five tips that can help you use technology to address security challenges:

  1. Randomize: Shuffling the order of the choices can help protect the security of the assessment. The questions can also be delivered in a random order themselves — to help prevent cribbing when users are sitting in non-screened assessment centers.
  2. Encrypt: With so many tests and exams being delivered via the Internet or an intranet, encryption can protect against interception. A Secure Socket layer (SSL) is a protocol that allows the browser and web server to encrypt their communication; anyone intercepting the communication can’t read it.
  3. Schedule: You can discourage cheating by specifying user names and passwords, setting assessment start times, limiting the length of time for an assessment and the number of times it may be taken.
  4. Monitor: A participant can’t start a monitored assessment until a proctor or invigilator has logged on to verify the participant’s identity. The monitor can be limited to a range of IP addresses to ensure that a certain physical location is used to administer the assessment.
  5. Secure browsers: It is possible to ‘lock down’ computers to keep participants from accessing other applications and websites while taking a medium- or high-stakes assessment. A secure browser prevents candidates from printing, capturing screens, accidentally exiting the assessment viewing source, task switching, etc.

Want more info? Download the White Paper: Delivering Assessments Safely and Securely  [registration required]