Test Security: Not Necessarily a Question of Proctoring Mode

Austin Fossey-42Posted by Austin Fossey

I recently spent time looking for research studies that analyzed the security levels of online and in-person proctoring. Unfortunately, no one seems to have compared these two approaches with a well-designed study. (If someone has done a rigorous study contrasting these two modes of delivery, please let me know! I certainly may have overlooked it in my research.)

I did learn a lot from the sparse literature that was available, and my main takeaway is this: security is related less to proctoring mode than it is to how much effort the test developer puts into administration planning and test design. Investing in solid administration policies, high-quality monitoring technology, and well-trained proctors is what really matters most for both in-person and online proctoring.

With some effort, testing programs with online proctors can likely achieve levels of security and service comparable to the services offered by many test centers. This came into focus for me after attending several recent seminars about online and in-person proctoring through the Association of Test Publishers (ATP) and Performance Testing Council (PTC).

The Standards for Educational and Psychological Testing provide a full list of considerations for organizations running any type of exam, but here are a few key points gleaned from the Standards and from PTC’s webinar (.wmv) to help you plan for online proctoring:

Control of the Environment

Unless a collaborator is onsite to set up and maintain the test environment, all security controls will need to be managed remotely. Here are suggestions for what you would need to do if you were a test program administrator under those circumstances:

  • Work with your online proctors to define the rules for acceptable test environments.
  • Ensure that test environment requirements are realistic for participants while still meeting your standards for security and comparability between administrations.
  • If security needs demand it, have monitoring equipment sent in advance (e.g., multiple cameras for improved monitoring, scanners to authenticate identification).
  • Clearly communicate policies to participants and get confirmation that they understand and can abide by your policies.
  • Plan policies for scenarios that might arise in an environment that is not managed by the test program administrator or proctor. For example, are you legally allowed to video someone who passes by in the background if they have not given their permission to be recorded? If not, have a policy in place stating that the participant is responsible for finding an isolated place to test. Do you or the proctoring company manage the location where the test is being delivered? If not, have a policy for who takes responsibility and absorbs the cost of an unexpected interruption like a fire alarm or power outage.

You should be prepared to document the comparability of administrations. This might include describing potential variations in the remote environment and how they may or may not impact the assessment results and security.

It is also advisable to audit some administrations to make sure that the testing environments comply with your testing program’s security policy. The online proctors’ incident reports should also be recorded in an administration report, just as they would with an in-person proctor.

Test Materials

You also need to make sure that everything needed to administer the test is provided, either physically or virtually.

  • Each participant must have the equipment and resources needed to take the test. If it is not reasonable to expect the participant to handle these tasks, you need to plan for someone else to do so, just as you would at a test center. For example, it might not be reasonable to expect some participant populations to know how to check whether the computer used for testing meets minimum software requirements.
  • If certain hardware (e.g., secured computers, cameras, scanners, microphones) or test materials (e.g., authorized references, scratch paper) are needed for the assessment design, you need to make sure these are available onsite for the participant and make sure they are collected afterwards.

Accommodations

Accommodations may take the form of physical or virtual test materials, but accommodations can also include additional services or some changes in the format of the assessment.

  • Some accommodations (e.g., extra time, large print) can be controlled by the assessment instrument or an online proctor, just as they would in a test center.
  • Other accommodations require special equipment or personnel onsite. Some personnel (e.g., scribes) may be able to provide their services remotely, but accommodations like tactile printouts of figures for the blind must be present onsite.

Extra effort is clearly needed when setting up an online-proctored test. Activities that might have been handled by a testing center (control of the environment, management of test materials, providing accommodations) now need to be remotely coordinated by the test program staff and proctors; however, the payoffs may be worth the extra effort. If comparable administration practices can be achieved, online-proctored assessments may be cheaper than test centers, offer increased access to participants, and lower the risks of collaborative cheating.

For more on online proctoring, check out this informational page and video below

ATP Highlights: Security and Future Innovations

Jim Farrell HeadshotPosted by Jim Farrell

One of the early highlights of every year for a lot of us here at Questionmark is the Association of Test Publishers (ATP) Conference – Innovations in Testing.

Established in 1992, ATP is a non-profit organization representing providers of tests and assessment tools and/or services related to assessment, selection, screening, certification, licensing, educational or clinical uses. The conference offers its members the opportunity for networking, workshops and sessions led by industry leaders. This year’s conference was held in Ft. Lauderdale, Florida (not a bad place to be in early February) and o had record attendance.

For me, there were two themes that stood out: Security and Looking to the Future.

ericatATPSecurity is always paramount at Questionmark. We often describe the issues in higher stakes tests with the Fraud Triangle (Rationalization, Opportunity, and Motivation) and the actual threats (Impersonation, Content Theft and Cheating). As shown in the picture to the left, our CEO, Eric Shepherd led a panel on remote monitoring with our some of our good friends including Don Kassner of ProctorU, Doug Winneg of Software Secure and Ruben Garcia of Innovative Exams. Each company provides a different level of security including recorded video of the test taker (Software Secure), Live Proctoring (ProctorU) and a Secure Kiosk with Live Proctoring (Innovative Exams). At Questionmark we see it as a sliding scale.

stakes ATP As testing moves from brick and mortar test centers to community centers and libraries, remote proctoring is becoming a real solution and Questionmark is excited to be part of it.

The other interesting trend at this conference was “the future”. The keynote speaker was Jack Uldrich, and he is a futurist. What does a futurist do, you ask? Here is a video that shows some of Jack Uldrich’s books and ideas. He says that “what we don’t know yet is just as important as what we know today. In this unknown knowledge is extraordinary opportunity.”

I really love this quote because it is a direct challenge to be innovative and unlearn things that were once true but are not true for the future. We have to realize that we are going to do some of the same things we have always done, but we are going to do them differently. This is either inherently scary or extremely exciting. I believe it is the latter, and that it’s up to us to listen to the trends and always have our eye on what is not yet possible.

I am looking forward to having conversations about the future of testing with our customers at the 2013 Questionmark User’s Conference in Baltimore, March 3-6. Click here to register.

International certifications: To translate or not to translate?

Sue Orchard

Posted by Joan Phaup

Scoring techniques, test delivery options, item generation and the intricacies of translating tests into different languages were among the many subjects covered during the Association of Test Publishers’ Innovations in Testing Conference last month.

Curious to know in particular about issues relating to test translation and localization, I spoke briefly with  Sue Orchard of Comms Multilingual about her perspective on the conference:

Were there any particular themes that emerged about translation and localization?

More and more organizations in North America are looking at taking their certifications international. One of the main themes is whether these certifications and any related training and marketing materials need to be translated or not. Some organizations have decided to leave their materials in English. My response to that would be: Are you testing people’s knowledge, skills and abilities or are you testing their knowledge of the English language?

What are the key elements that make for a high-quality translation?

Preparation is absolutely key in ensuring a successful outcome. When creating exams, tests and assessments in the first place, it is important to write these with translation in mind. You should avoid jargon, complicated sentences, overcrowding of the text on a page and many other things. If the exams, tests and assessments have not been created with translation in mind, then this can cause problems during a translation project.

What do you look for in validating a translation?

It is very important to follow specific process steps to ensure the validation of a translation. The actual steps to be taken will vary from client to client, depending on their own capabilities, such as the availability of native-speaker Subject Matter Experts. When translating, localizing and adapting exams, tests and assessments, the steps to be taken will require much more work than for the translation of training or marketing materials, which just require translation into the language and proof-reading.

Going forward, what do you see as the key issues organization will face as they continue to expand their international and intercultural testing programs?

There are many issues that need to be considered by organizations that are looking to expand internationally. Should the exams, tests and assessments be left in English or translated? What about related materials such as training or marketing materials?

Should the certification remain exactly as it is in the original country, or should organizations attempt to get the certification licensed in the target market? Is the exam, test or other assessment culturally valid in the target country? Can it be localized and adapted or is it not suitable at all for people in other countries?

For more on this subject see the Q&A at the end of my previous post about Sue’s February 16 Questionmark web seminar on assessment translation and localization.

The Value of Our Community

jim_small Posted by Jim Farrell

If you search on the Internet for the term “community” you get a tremendous number of hits, from Wikipedia definitions to the funny television show (not a commercial) with that title. When you go to the Wikipedia article, there are two things that stand out: 1. The definition for the word community is debated, so much so that by the 1950s there were 94 discrete definitions! 2: The advent of the Internet has changed the concept of community by removing the tie to a geographic location and focusing on common interests. That is the type of community I want to talk about.

globeAt the Association of Test Publishers’ (ATP)  Innovations in Testing Conference in Arizona last week, I sat in on a very interesting session titled “The Wild and Wacky World of Customer Communities” led by members of the Microsoft Certification Testing team. The goal of the session was to help people start customer communities in order to improve relationships with stakeholders — in their case the test takers themselves. I found it interesting because many of the issues people were facing, we at Questionmark had faced or continue to face today in our own customer community. The discussion was very engaging and people there definitely saw the advantages of sponsoring a community.

For those of you who don’t know, we rebuilt our online Questionmark Community last year. Our new Questionmark Community Spaces is a much more robust place for customers to ask questions, interact with other customers  and communicate with product managers. Some of the most authentic conversations  I witness come from customers answering questions posted by other customers. Product managers like me use the community space to post roadmaps, answer customer questions and discuss upcoming releases and features. We see it as a way to keep in touch with as many customers as possible and gauge their responses to new ideas. I’m also planning to post more videos there that show people how to use features  —  something I oh so love to do!

If you are a Questionmark software support plan customer and have not signed up to be a member of our Community Spaces I highly encourage you to do so, and I will see you there. Just go  to our Web site and click on Community Spaces under the “My Questionmark” tab.

New Options for Delivering High-Stakes Exams Securely

 

Posted by Joan Phaup

We always look forward to the Association of Test Publishers’ Innovations in Testing Conference – a great place to stay up to date with all the changes happening in the testing and assessment industry and to share some of our own recent progress.

At this year’s conference in Phoenix next week we’ll be involved in a number of presentations, one of which will focus on new options for delivering high-stakes exams.

Our CEO, Eric Shepherd, will be sharing the podium with Allison Horn from Accenture and Ruben Garcia from Innovative Exams to talk about the use of remotely monitored testing stations as a secure but flexible alternative when it’s too impractical or costly to schedule locally proctored exams. They’ll explore how remote proctoring compares to in-person proctoring, talk about how participants in remotely proctored tests can be authenticated and discuss ways to protect exam content and prevent cheating. They’ll also consider what circumstances lend themselves most readily to the use of self-service testing stations.

Eric has given a lot of thought to the issues surrounding secure test delivery and mentions the test kiosk option in a wide-ranging article on Oversight, Monitoring and Delivery of Higher Stakes Assessments Safely and Securely – a recommended read!