Test Security: Not Necessarily a Question of Proctoring Mode

Austin Fossey-42Posted by Austin Fossey

I recently spent time looking for research studies that analyzed the security levels of online and in-person proctoring. Unfortunately, no one seems to have compared these two approaches with a well-designed study. (If someone has done a rigorous study contrasting these two modes of delivery, please let me know! I certainly may have overlooked it in my research.)

I did learn a lot from the sparse literature that was available, and my main takeaway is this: security is related less to proctoring mode than it is to how much effort the test developer puts into administration planning and test design. Investing in solid administration policies, high-quality monitoring technology, and well-trained proctors is what really matters most for both in-person and online proctoring.

With some effort, testing programs with online proctors can likely achieve levels of security and service comparable to the services offered by many test centers. This came into focus for me after attending several recent seminars about online and in-person proctoring through the Association of Test Publishers (ATP) and Performance Testing Council (PTC).

The Standards for Educational and Psychological Testing provide a full list of considerations for organizations running any type of exam, but here are a few key points gleaned from the Standards and from PTC’s webinar (.wmv) to help you plan for online proctoring:

Control of the Environment

Unless a collaborator is onsite to set up and maintain the test environment, all security controls will need to be managed remotely. Here are suggestions for what you would need to do if you were a test program administrator under those circumstances:

  • Work with your online proctors to define the rules for acceptable test environments.
  • Ensure that test environment requirements are realistic for participants while still meeting your standards for security and comparability between administrations.
  • If security needs demand it, have monitoring equipment sent in advance (e.g., multiple cameras for improved monitoring, scanners to authenticate identification).
  • Clearly communicate policies to participants and get confirmation that they understand and can abide by your policies.
  • Plan policies for scenarios that might arise in an environment that is not managed by the test program administrator or proctor. For example, are you legally allowed to video someone who passes by in the background if they have not given their permission to be recorded? If not, have a policy in place stating that the participant is responsible for finding an isolated place to test. Do you or the proctoring company manage the location where the test is being delivered? If not, have a policy for who takes responsibility and absorbs the cost of an unexpected interruption like a fire alarm or power outage.

You should be prepared to document the comparability of administrations. This might include describing potential variations in the remote environment and how they may or may not impact the assessment results and security.

It is also advisable to audit some administrations to make sure that the testing environments comply with your testing program’s security policy. The online proctors’ incident reports should also be recorded in an administration report, just as they would with an in-person proctor.

Test Materials

You also need to make sure that everything needed to administer the test is provided, either physically or virtually.

  • Each participant must have the equipment and resources needed to take the test. If it is not reasonable to expect the participant to handle these tasks, you need to plan for someone else to do so, just as you would at a test center. For example, it might not be reasonable to expect some participant populations to know how to check whether the computer used for testing meets minimum software requirements.
  • If certain hardware (e.g., secured computers, cameras, scanners, microphones) or test materials (e.g., authorized references, scratch paper) are needed for the assessment design, you need to make sure these are available onsite for the participant and make sure they are collected afterwards.

Accommodations

Accommodations may take the form of physical or virtual test materials, but accommodations can also include additional services or some changes in the format of the assessment.

  • Some accommodations (e.g., extra time, large print) can be controlled by the assessment instrument or an online proctor, just as they would in a test center.
  • Other accommodations require special equipment or personnel onsite. Some personnel (e.g., scribes) may be able to provide their services remotely, but accommodations like tactile printouts of figures for the blind must be present onsite.

Extra effort is clearly needed when setting up an online-proctored test. Activities that might have been handled by a testing center (control of the environment, management of test materials, providing accommodations) now need to be remotely coordinated by the test program staff and proctors; however, the payoffs may be worth the extra effort. If comparable administration practices can be achieved, online-proctored assessments may be cheaper than test centers, offer increased access to participants, and lower the risks of collaborative cheating.

For more on online proctoring, check out this informational page and video below

Online or test center proctoring: Which is more secure?

John Kleeman HeadshotPosted by John Kleeman

As explained in my previous post Online or test center proctoring: Which is best?, a new way of proctoring certification exams is rapidly gaining traction. With online proctoring, candidates takes exams at their office or home, with a proctor observing via video camera over the Internet. Parents scaling the walls of a building to help their children cheat

The huge advantage of online proctoring is that the candidate doesn’t need to travel to a test center. This is fairer and saves a lot of time and cost. But how secure is online proctoring? You might at first sight think that test center proctoring is more secure – as it sounds easier to spot cheating in a controlled environment and face-to-face than online. But it’s not as simple as that.

The stakes for a candidate to pass an exam are often high, and there are many examples where proctors at test centers coach candidates or otherwise breach the integrity of the exams process. A proctor in a test center can witness the same test being taken over and over again, and they can start to memorize, and potentially sell, the content that they see.  For example, according to a 2011 article in the Economist , one major test center company at that time was shutting down five test centers a week due to security concerns.

Test center vulnerabilities are not always as obvious as they are in the picture to the right (source here), but they are myriad. This recent photo shows parents in India climbing the walls of a building to help their children pass exams, with proctors bribed to help. According to Standard Digital:

“Supervisors stationed at notorious test centres vie for the postings, enticed by the prospect of bribes from parents eager to have their wards scrape through.”

Proxy test taking – where one person takes a test impersonating another – is a also big concern in the industry. A 2014 Computer World article quotes an expert saying:

“In some cases, proxies have been able to skirt security protocols by visiting corrupt testing facilities overseas that operate both a legitimate “front room” test area and a fraudulent “back room” operation.

This doesn’t just happen in a few parts of the world: there are examples worldwide. For instance, there was a prominent case in the UK in 2014 where proctors were dishonest in a test used to check English knowledge for candidates seeking visas. According to a BBC report, in some tests the proctor read out the correct answers to candidates. And in another test, a candidate came to the test center and had their picture taken, but then a false sitter went on to take the test. An undercover investigator posing as a candidate was told:

“Someone else will sit the exam for you. But you will have to have your photo taken there to prove you were present.”

This wasn’t a small scale affair – the UK government announced that at least 29,000 exam results were invalid due to this fraud.

Corrupt test centers have also been found in the US. In May 2015, a New York man was sentenced to jail for being involved in fraud where five New York test centers allowed applicants for a commercial driving license to pay their way to pass the test. According to a newspaper report:

“The guards are accused of taking bribes to arrange for customers to leave the testing room with their exams, which they gave to a surrogate test-taker outside who looked up the answers on a laptop computer. The guards would allow the test-takers to enter and leave the testing rooms.”

There are many other examples of this kind of cheating at test centers – a good source of information is  Caveon’s blog about cheating in the news. Caveon and Questionmark recently announced a partnership to generally enhance the security of high-stakes testing programs. The partnership with Caveon will also provide Questionmark’s customers with easy access to consulting services to help them enhance the security of the exams.
Of course, most test center proctors are honest and most test center exams are fair, but there are enough problems to raise concerns. Online proctoring has some security disadvantages, too:

  • Although improvements are being developed, it is harder for the proctor to check whether an ID is genuine when looking at it through a camera.
  • A remote camera in the candidate’s own environment is less capable of spotting some forms of cheating than a controlled environment in a test center.

But there are also genuine security advantages.  It is much harder for an online proctor to get to know a candidate to be able to coach him or her or receive a payment to help in other ways.

  • Because proctors can be assigned randomly and without any geographic connection, it’s much less likely for the proctor and candidate to be able to pre-arrange any bad behavior
  • All communication between proctor and candidate is electronic and can be logged, so the candidate cannot easily make an inappropriate approach during the exam.
  • While test center proctors have easy access to exam content which can lead to various types of security breaches, online proctors can be restricted from viewing the exam content through the use of such technologies as secure browsers.
  • Because there is less difficulty and cost involved in online proctoring than when the candidate travels to a physical test center, it’s practical to test more frequently– and this is a security benefit. If there is frequent testing, it may be simpler for a candidate to learn the material and pass the test honestly than put a lot of effort into cheating. If you have several exams, you can also compare the pictures of a candidate at each exam to reduce the chance of impersonation.

In summary, the main reason for online proctoring is that it saves time and money over going to a bricks-and-mortar test center. The security advantages and disadvantages of test center versus online proctoring  are open to debate.  Dealing with security vulnerabilities requires constant vigilance. With new online proctoring technologies enhancing exam security, many certification programs are now transitioning away from test centers. Traditionally a test center was a secure place to administer exams, but in practice there have been so many incidents of proctor dishonesty over the years that online proctoring is likely justifiable for security reasons.

Online or test center proctoring: Which is best?

John Kleeman HeadshotPosted by John Kleeman

A new way of proctoring certification exams is rapidly gaining traction. This article compares and contrasts the old with the new.

Many high-tech companies offer certification exams for consultants, users and implementers. Such exams often require candidates to travel to a bricks-and-mortar test center where proctors (or invigilators) supervise the process.

Now, however, online proctoring is becoming prevalent: each candidate takes the exams at his or her home or office, with a proctor observing via video camera over the Internet. Two of the world’s largest software companies, SAP and Microsoft, offer online proctoring for their certification programs, and many other companies are looking to follow suit. This article explains some of the pros and cons of the two approaches.

workplace_addFactors for choosing online proctoring

  • Reduced travel time.  Candidates can take an exam without wasting time traveling to a test center. This is an important saving for their employers – often the test sponsor’s customers.
  • Convenient scheduling. A candidate can choose a convenient time, for example after the kids have gone to bed or when work pressures are lowest. Usually one needs to book in advance to attend a test center, but it’s often possible to schedule an online proctor at short notice.
  • Fairness. With an exam at a test center, some people will have had a short journey and others a longer one. Some might have experienced a traffic jam or other hassle getting there. This gives an advantage to those who happen to live closer, as they will have less anxiety. An online experience reduces the variability of the exam experience.
  • Accessibility. Candidates take online proctored exams on their own computers, using their normal accessibility aids such as screen readers or special input devices, whereas these require setup at a test center. Some test centers only provide their own (often limited) tools for providing accommodations, so candidates are working with unfamiliar tool sets. This places them at a disadvantage. Also, for people with certain disabilities, travel is a major inconvenience.
  •  Keeping certifications up to date. If candidates have to travel to a test enter, a test sponsor can’t realistically require an exam to be taken more than once every few years. But in today’s world, products and job skills change very quickly, so certification risks being out of date. The availability of online  proctoring allows update exams (assessing candidates on what has changed since their last exam) to be taken as products change, which makes the programme more valid.
  • Greater authenticity. The more authentic assessments are, the more they measure actual performance. See Will Thalheimer’s excellent paper on measuring learning results for more on this. Assessing someone in their work environment with online proctoring is more authentic and so will likely measure performance better than putting them in a test center.

office-buildingFactors for choosing test center proctoring

  • Standardized computers. While online proctoring requires the candidate to have an appropriate computer, internet connection and webcam that they know how to use, test centers provide a computer that is already set up. For most certification programmes, it’s fair and reasonable that candidates use their own computers (often called BYOD – Bring Your Own Device). But for some programmes, this might be less fair. For example, in professions where IT literacy is not required, it might not be fair to expect people to have access to a PC with webcam that they know how to use.
  • Very long exams.  In online-proctored exams, the candidate is usually forbidden from taking a break for security reasons.  Most exams can be taken in one sitting, but if exams are longer than three hours, a test center makes sense.
  • Regulation. Some regulators or government authorities may require delivery of an exam with a physically present proctor at a test center.
  • Geographical convenience.  In some cases, test centers may be close at hand. For example, a university might have all its candidates already present, or, for some test sponsors, candidates may all live in metropolitan areas close to test centers.

checkOther factors to consider:

  • Language. In theory, a candidate could schedule an online proctor in his or her own language, though in practice many programs only offer English-speaking proctors. A test center may well not have proctors who can speak different languages, but typically will speak the local language.
  • Security. You might think that the security is stronger in a test center than with online proctoring. However, over the years there have been many incidents where face-to-face proctors have coached candidates. Online proctoring also makes it feasible to administer exams more frequently, which helps security by making impersonation harder. This is a big subject, and I’ll follow up with a blog post about security.

I’d welcome your thoughts on any other factors for and against online proctoring.

Case Study: Live monitoring offers security for online tests

Headshot JuliePosted by Julie Delazyn

Thomas Edison State College (TESC) is one of the oldest schools in the country designed specifically for adults. The college’s 20,000+ students, many of them involved with careers and families, live all over the world and favor courses that enable online study.

In setting up online midterm and final exams, the college wanted to give distance leaners the same kind of security as on-campus students experience at more traditional institutions. At the same time, it was essential to give students some control over where and when they take tests.

Online proctoring offered a way to achieve both of these goals.

Working with Questionmark and ProctorU has enabled TESC to administer proctored exams to students at their home or work computers.

Proctors connect with test takers via webcam and audio hook-ups, verify the each test-taker’s identity, initiate the authentication process, ensure the students are not using any unauthorized materials or aids and troubleshoot technical problems. The college can now run secure tests while meeting the needs of busy students for flexible access to exams.

You can read the full case study here.