Will 2016 mark the end of test centers?

John Kleeman HeadshotPosted by John Kleeman

The Internet is causing disruptive change in many industries. When I was a child, we had a bookshelf full of encyclopaedias, nowadays my children can find out much more than was ever in these large books on Google or Wikipedia. This is just one of many disruptions the Internet has brought.

Building being demolishedCurrently many high-stakes exams are taken at “bricks and mortar” test centers. Participants need to travel to a test center to take the exam, often having to journey a considerable distance; tests need to be scheduled for a fixed slot in advance; and participants or their employer must pay a substantial fee for use of the test center. Increasingly, the Internet is allowing a new dynamic, with participants taking tests at their offices or homes, with a remote, online proctor ensuring integrity.

So will the Internet take another scalp? Will 2016 be the year that test centers for high stakes tests and exams start to disappear? Will there even be bulldozers demolishing test centers or re-purposing them for other tasks?

Interested in learning more about online proctoring? I will be presenting a session on Ensuring Exam Integrity with Online Proctoring at the Questionmark Conference 2016: Shaping the Future of Assessment in Miami, April 12-15. I look forward to seeing you there! Click here to register and learn more about this important learning event. 

It may be a while before the bulldozers move in, but as shown in my previous post Online or test center proctoring: Which is best?, there are many advantages of online proctoring. Questionmark exams can be delivered both at test centers and via online proctoring, but I am increasingly coming to believe that online proctoring is the future for high stakes exams.

The benefit to the participant of not having to travel to a test center and being able to take the exam in a flexible timeslot is immense. To illustrate this, here is a quote from an independent blog entry by a candidate describing his experience taking an SAP certification exam in 2015:

“I was able to take my test from my home office. This is such a big improvement because I did not have to drive to a test centre which in my case is far away and where I can only take the test on specific days in the year.”

The main reason that organizations use test centers is that it feels more secure. But as I suggested in Online or test center proctoring: Which is more secure?, there are arguments on both sides for security, and most organizations probably use test centers rather than online proctoring because it’s what they have been doing for a long time, rather than necessarily because it is safer.

Undercover picture fromm BBC of test cheatingThere is a long history of security breaches at test centers, and some of the most flagrant happen due to proctors or invigilators helping participants pass the test. The BBC recently reported on several cases of cheating within construction industry exams at test centers in the UK – see here for a text summary and here for an excellent 13 minute video including undercover footage of the cheating. In one case, a proctor took the exam for the participant; in another case, a roomful of people were taking the exam and someone was showing the answers on a large screen at the front of the room (see picture to the right); and in a third case, participants answered the exam but didn’t submit, and the proctor reviewed and improved the answers before submitting.

This is just one of many, many examples. When participants see a financial benefit to passing a high stakes test, there are challenges in maintaining the integrity of a face-to-face proctoring system. It’s not impossible for online proctors to be corrupt, but when separated by distance, it is much harder.

The testing industry is very conservative. There will still be exams taken in test centers for many years to come. But I do think that 2016 may well be the beginning of the end for “bricks and mortar” test centers — and that during 2016, the mainstream view will start to think that online proctoring is the future.

Interested in learning more about online proctoring? I will be presenting a session on Ensuring Exam Integrity with Online Proctoring at the Questionmark Conference 2016: Shaping the Future of Assessment in Miami, April 12-15. I look forward to seeing you there! Click here to register and learn more about this important learning event. 

For more on online proctoring, check out this informational page and video below:

Scary Findings : Proctors often involved in test-center cheating

John Kleeman HeadshotPosted by John Kleeman

Over Halloween, I’ve been reviewing how often it seems that test center administrators or proctors have been shown to help candidates cheat at exams. It’s scary how often this appears to happen.

BBC undercover pictureJust a couple of weeks ago, a BBC television investigation reported widespread cheating at UK test centers where construction workers and builders were certified on health and safety. The BBC showed (see undercover picture to the right) a test center director reading exam answers from a big screen, instructing candidates:

“Follow me on screen, guys. I’m going to shout the correct answer, you just click. We’re going to make a couple of mistakes – what I don’t want is everyone making the same mistake.”

The sad thing is that construction is a dangerous occupation. In the past five years, the BBC reports that 221 workers died in the UK while on the job within the construction sector. It’s very worrying that corrupt test centers that facilitate cheating on health and safety tests are likely contributing to this.

Another scary example is from a recent US court case where a decorated former police officer in San Francisco was sentenced to two years in jail for taking bribes from taxi drivers to give them a passing grade, whether or not they passed the test. These are a couple of examples I happen to have seen this weekend. See my earlier blog entry Online or test center proctoring: Which is more secure? for several other examples of test center fraud.

So what is the answer?  Part of the solution as I argued in What is the best way to reduce cheating? is to remove people’s rationalization to cheat. Most people think of themselves as good, honest people, and if you communicate the aims of the test and take other measures to make people think the test is fair, then fewer of them are likely to cheat.

Another approach is to do what Cambodia has been doing and throw a lot of resources into preventing cheating. According to this article, the government’s anti-corruption unit has been focusing on university exams, enlisting 2,000 volunteers to help monitor last summer’s exams and prevent collusion between proctors and students.

Of course, the vast majority of tests at test centers are entirely legitimate, and reputable test center providers do all they can to prevent face-to-face proctors from colluding with candidates. But there does seem to be two persistent problems:

  1. Some proctors are keen to help their local candidates
  2. The financial stakes involved in passing a test means that when candidate and proctor meet face-to-face, there is an ever-present risk of corruption.

I strongly suspect online proctoring is part of the solution here. The main argument for online proctoring is that candidates do not need to travel to a test center (see Online or test center proctoring: Which is best?). But there is an important side benefit to this: candidates and proctor never meet, and all their communications can be recorded. Without a face-to-face meeting and without a local connection, the likelihood of collusion, so this kind of cheating is much less probable. Now, that’s a non-scary solution that has some promise.

SAP to present their global certification program at London briefing

Chloe MendoncaPosted by Chloe Mendonca

A key to SAP’s success is ensuring that the professional learning path of skilled SAP practitioners is continually supported – thereby making qualified experts on their cloud solutions readily available to customers, partners and consultants.

In a world where current knowledge and skills are more important than ever, SAP needed a way to verify that their cloud consultants around the world were keeping their knowledge and skills up-to-date  with rapidly changing technology. A representative of the certification program at SAP comments:breakfast briefing

It became clear that a certification that lasted for two or three years didn’t cut it any longer – in all areas of the portfolio. Everything is evolving so quickly, and SAP has to always support current, validated knowledge.”

Best Practices from SAP

The move to the cloud required some fundamental changes to SAP’s existing certification program. What challenges did they face? What technologies are they using to ensure the security of the program? Join us on the 21st of October for a breakfast briefing in London, where Ralf Kirchgaessner, Manager of Global Certification at SAP, will discuss the answers to these questions. Ralf will tell how the SAP team planned for the program, explain its benefits and share lessons learned.

Click here to learn more and register for this complimentary breakfast briefing *Seats are limited

High-Stakes Assessments

The briefing will  include a best-practice seminar on the types of technologies and techniques to consider using as part of your assessment program to securely create, deliver and report on high-stakes tests around the world. It will highlight technologies such as online invigilation, secure browsers and item banking tools that alleviate the testing centre burden and allow organisations and test publishers to securely administer trustable tests and exams and protect valuable assessment content.

What’s a breakfast briefing?

You can expect a morning of networking, best practice tips and live demonstrations of the newest assessment technologies.The event will include a complimentary breakfast at 8:45 a.m. followed by presentations and discussions until about 12:30 p.m.

Who should attend?

These gatherings are ideal for people involved in certification, compliance and/or risk management, and learning and development.

When? Where?

Wednesday 21st October at Microsoft’s Office in London, Victoria — 8:45 a.m. – 12:30 p.m

Click here to learn more and register to attend

Test Security: Not Necessarily a Question of Proctoring Mode

Austin Fossey-42Posted by Austin Fossey

I recently spent time looking for research studies that analyzed the security levels of online and in-person proctoring. Unfortunately, no one seems to have compared these two approaches with a well-designed study. (If someone has done a rigorous study contrasting these two modes of delivery, please let me know! I certainly may have overlooked it in my research.)

I did learn a lot from the sparse literature that was available, and my main takeaway is this: security is related less to proctoring mode than it is to how much effort the test developer puts into administration planning and test design. Investing in solid administration policies, high-quality monitoring technology, and well-trained proctors is what really matters most for both in-person and online proctoring.

With some effort, testing programs with online proctors can likely achieve levels of security and service comparable to the services offered by many test centers. This came into focus for me after attending several recent seminars about online and in-person proctoring through the Association of Test Publishers (ATP) and Performance Testing Council (PTC).

The Standards for Educational and Psychological Testing provide a full list of considerations for organizations running any type of exam, but here are a few key points gleaned from the Standards and from PTC’s webinar (.wmv) to help you plan for online proctoring:

Control of the Environment

Unless a collaborator is onsite to set up and maintain the test environment, all security controls will need to be managed remotely. Here are suggestions for what you would need to do if you were a test program administrator under those circumstances:

  • Work with your online proctors to define the rules for acceptable test environments.
  • Ensure that test environment requirements are realistic for participants while still meeting your standards for security and comparability between administrations.
  • If security needs demand it, have monitoring equipment sent in advance (e.g., multiple cameras for improved monitoring, scanners to authenticate identification).
  • Clearly communicate policies to participants and get confirmation that they understand and can abide by your policies.
  • Plan policies for scenarios that might arise in an environment that is not managed by the test program administrator or proctor. For example, are you legally allowed to video someone who passes by in the background if they have not given their permission to be recorded? If not, have a policy in place stating that the participant is responsible for finding an isolated place to test. Do you or the proctoring company manage the location where the test is being delivered? If not, have a policy for who takes responsibility and absorbs the cost of an unexpected interruption like a fire alarm or power outage.

You should be prepared to document the comparability of administrations. This might include describing potential variations in the remote environment and how they may or may not impact the assessment results and security.

It is also advisable to audit some administrations to make sure that the testing environments comply with your testing program’s security policy. The online proctors’ incident reports should also be recorded in an administration report, just as they would with an in-person proctor.

Test Materials

You also need to make sure that everything needed to administer the test is provided, either physically or virtually.

  • Each participant must have the equipment and resources needed to take the test. If it is not reasonable to expect the participant to handle these tasks, you need to plan for someone else to do so, just as you would at a test center. For example, it might not be reasonable to expect some participant populations to know how to check whether the computer used for testing meets minimum software requirements.
  • If certain hardware (e.g., secured computers, cameras, scanners, microphones) or test materials (e.g., authorized references, scratch paper) are needed for the assessment design, you need to make sure these are available onsite for the participant and make sure they are collected afterwards.

Accommodations

Accommodations may take the form of physical or virtual test materials, but accommodations can also include additional services or some changes in the format of the assessment.

  • Some accommodations (e.g., extra time, large print) can be controlled by the assessment instrument or an online proctor, just as they would in a test center.
  • Other accommodations require special equipment or personnel onsite. Some personnel (e.g., scribes) may be able to provide their services remotely, but accommodations like tactile printouts of figures for the blind must be present onsite.

Extra effort is clearly needed when setting up an online-proctored test. Activities that might have been handled by a testing center (control of the environment, management of test materials, providing accommodations) now need to be remotely coordinated by the test program staff and proctors; however, the payoffs may be worth the extra effort. If comparable administration practices can be achieved, online-proctored assessments may be cheaper than test centers, offer increased access to participants, and lower the risks of collaborative cheating.

For more on online proctoring, check out this informational page and video below

Online or test center proctoring: Which is more secure?

John Kleeman HeadshotPosted by John Kleeman

As explained in my previous post Online or test center proctoring: Which is best?, a new way of proctoring certification exams is rapidly gaining traction. With online proctoring, candidates takes exams at their office or home, with a proctor observing via video camera over the Internet. Parents scaling the walls of a building to help their children cheat

The huge advantage of online proctoring is that the candidate doesn’t need to travel to a test center. This is fairer and saves a lot of time and cost. But how secure is online proctoring? You might at first sight think that test center proctoring is more secure – as it sounds easier to spot cheating in a controlled environment and face-to-face than online. But it’s not as simple as that.

The stakes for a candidate to pass an exam are often high, and there are many examples where proctors at test centers coach candidates or otherwise breach the integrity of the exams process. A proctor in a test center can witness the same test being taken over and over again, and they can start to memorize, and potentially sell, the content that they see.  For example, according to a 2011 article in the Economist , one major test center company at that time was shutting down five test centers a week due to security concerns.

Test center vulnerabilities are not always as obvious as they are in the picture to the right (source here), but they are myriad. This recent photo shows parents in India climbing the walls of a building to help their children pass exams, with proctors bribed to help. According to Standard Digital:

“Supervisors stationed at notorious test centres vie for the postings, enticed by the prospect of bribes from parents eager to have their wards scrape through.”

Proxy test taking – where one person takes a test impersonating another – is a also big concern in the industry. A 2014 Computer World article quotes an expert saying:

“In some cases, proxies have been able to skirt security protocols by visiting corrupt testing facilities overseas that operate both a legitimate “front room” test area and a fraudulent “back room” operation.

This doesn’t just happen in a few parts of the world: there are examples worldwide. For instance, there was a prominent case in the UK in 2014 where proctors were dishonest in a test used to check English knowledge for candidates seeking visas. According to a BBC report, in some tests the proctor read out the correct answers to candidates. And in another test, a candidate came to the test center and had their picture taken, but then a false sitter went on to take the test. An undercover investigator posing as a candidate was told:

“Someone else will sit the exam for you. But you will have to have your photo taken there to prove you were present.”

This wasn’t a small scale affair – the UK government announced that at least 29,000 exam results were invalid due to this fraud.

Corrupt test centers have also been found in the US. In May 2015, a New York man was sentenced to jail for being involved in fraud where five New York test centers allowed applicants for a commercial driving license to pay their way to pass the test. According to a newspaper report:

“The guards are accused of taking bribes to arrange for customers to leave the testing room with their exams, which they gave to a surrogate test-taker outside who looked up the answers on a laptop computer. The guards would allow the test-takers to enter and leave the testing rooms.”

There are many other examples of this kind of cheating at test centers – a good source of information is  Caveon’s blog about cheating in the news. Caveon and Questionmark recently announced a partnership to generally enhance the security of high-stakes testing programs. The partnership with Caveon will also provide Questionmark’s customers with easy access to consulting services to help them enhance the security of the exams.
Of course, most test center proctors are honest and most test center exams are fair, but there are enough problems to raise concerns. Online proctoring has some security disadvantages, too:

  • Although improvements are being developed, it is harder for the proctor to check whether an ID is genuine when looking at it through a camera.
  • A remote camera in the candidate’s own environment is less capable of spotting some forms of cheating than a controlled environment in a test center.

But there are also genuine security advantages.  It is much harder for an online proctor to get to know a candidate to be able to coach him or her or receive a payment to help in other ways.

  • Because proctors can be assigned randomly and without any geographic connection, it’s much less likely for the proctor and candidate to be able to pre-arrange any bad behavior
  • All communication between proctor and candidate is electronic and can be logged, so the candidate cannot easily make an inappropriate approach during the exam.
  • While test center proctors have easy access to exam content which can lead to various types of security breaches, online proctors can be restricted from viewing the exam content through the use of such technologies as secure browsers.
  • Because there is less difficulty and cost involved in online proctoring than when the candidate travels to a physical test center, it’s practical to test more frequently– and this is a security benefit. If there is frequent testing, it may be simpler for a candidate to learn the material and pass the test honestly than put a lot of effort into cheating. If you have several exams, you can also compare the pictures of a candidate at each exam to reduce the chance of impersonation.

In summary, the main reason for online proctoring is that it saves time and money over going to a bricks-and-mortar test center. The security advantages and disadvantages of test center versus online proctoring  are open to debate.  Dealing with security vulnerabilities requires constant vigilance. With new online proctoring technologies enhancing exam security, many certification programs are now transitioning away from test centers. Traditionally a test center was a secure place to administer exams, but in practice there have been so many incidents of proctor dishonesty over the years that online proctoring is likely justifiable for security reasons.

Online or test center proctoring: Which is best?

John Kleeman HeadshotPosted by John Kleeman

A new way of proctoring certification exams is rapidly gaining traction. This article compares and contrasts the old with the new.

Many high-tech companies offer certification exams for consultants, users and implementers. Such exams often require candidates to travel to a bricks-and-mortar test center where proctors (or invigilators) supervise the process.

Now, however, online proctoring is becoming prevalent: each candidate takes the exams at his or her home or office, with a proctor observing via video camera over the Internet. Two of the world’s largest software companies, SAP and Microsoft, offer online proctoring for their certification programs, and many other companies are looking to follow suit. This article explains some of the pros and cons of the two approaches.

workplace_addFactors for choosing online proctoring

  • Reduced travel time.  Candidates can take an exam without wasting time traveling to a test center. This is an important saving for their employers – often the test sponsor’s customers.
  • Convenient scheduling. A candidate can choose a convenient time, for example after the kids have gone to bed or when work pressures are lowest. Usually one needs to book in advance to attend a test center, but it’s often possible to schedule an online proctor at short notice.
  • Fairness. With an exam at a test center, some people will have had a short journey and others a longer one. Some might have experienced a traffic jam or other hassle getting there. This gives an advantage to those who happen to live closer, as they will have less anxiety. An online experience reduces the variability of the exam experience.
  • Accessibility. Candidates take online proctored exams on their own computers, using their normal accessibility aids such as screen readers or special input devices, whereas these require setup at a test center. Some test centers only provide their own (often limited) tools for providing accommodations, so candidates are working with unfamiliar tool sets. This places them at a disadvantage. Also, for people with certain disabilities, travel is a major inconvenience.
  •  Keeping certifications up to date. If candidates have to travel to a test enter, a test sponsor can’t realistically require an exam to be taken more than once every few years. But in today’s world, products and job skills change very quickly, so certification risks being out of date. The availability of online  proctoring allows update exams (assessing candidates on what has changed since their last exam) to be taken as products change, which makes the programme more valid.
  • Greater authenticity. The more authentic assessments are, the more they measure actual performance. See Will Thalheimer’s excellent paper on measuring learning results for more on this. Assessing someone in their work environment with online proctoring is more authentic and so will likely measure performance better than putting them in a test center.

office-buildingFactors for choosing test center proctoring

  • Standardized computers. While online proctoring requires the candidate to have an appropriate computer, internet connection and webcam that they know how to use, test centers provide a computer that is already set up. For most certification programmes, it’s fair and reasonable that candidates use their own computers (often called BYOD – Bring Your Own Device). But for some programmes, this might be less fair. For example, in professions where IT literacy is not required, it might not be fair to expect people to have access to a PC with webcam that they know how to use.
  • Very long exams.  In online-proctored exams, the candidate is usually forbidden from taking a break for security reasons.  Most exams can be taken in one sitting, but if exams are longer than three hours, a test center makes sense.
  • Regulation. Some regulators or government authorities may require delivery of an exam with a physically present proctor at a test center.
  • Geographical convenience.  In some cases, test centers may be close at hand. For example, a university might have all its candidates already present, or, for some test sponsors, candidates may all live in metropolitan areas close to test centers.

checkOther factors to consider:

  • Language. In theory, a candidate could schedule an online proctor in his or her own language, though in practice many programs only offer English-speaking proctors. A test center may well not have proctors who can speak different languages, but typically will speak the local language.
  • Security. You might think that the security is stronger in a test center than with online proctoring. However, over the years there have been many incidents where face-to-face proctors have coached candidates. Online proctoring also makes it feasible to administer exams more frequently, which helps security by making impersonation harder. This is a big subject, and I’ll follow up with a blog post about security.

I’d welcome your thoughts on any other factors for and against online proctoring.

« Previous PageNext Page »