I recently spent time looking for research studies that analyzed the security levels of online and in-person proctoring. Unfortunately, no one seems to have compared these two approaches with a well-designed study. (If someone has done a rigorous study contrasting these two modes of delivery, please let me know! I certainly may have overlooked it in my research.)
I did learn a lot from the sparse literature that was available, and my main takeaway is this: security is related less to proctoring mode than it is to how much effort the test developer puts into administration planning and test design. Investing in solid administration policies, high-quality monitoring technology, and well-trained proctors is what really matters most for both in-person and online proctoring.
With some effort, testing programs with online proctors can likely achieve levels of security and service comparable to the services offered by many test centers. This came into focus for me after attending several recent seminars about online and in-person proctoring through the Association of Test Publishers (ATP) and Performance Testing Council (PTC).
Unless a collaborator is onsite to set up and maintain the test environment, all security controls will need to be managed remotely. Here are suggestions for what you would need to do if you were a test program administrator under those circumstances:
Work with your online proctors to define the rules for acceptable test environments.
Ensure that test environment requirements are realistic for participants while still meeting your standards for security and comparability between administrations.
If security needs demand it, have monitoring equipment sent in advance (e.g., multiple cameras for improved monitoring, scanners to authenticate identification).
Clearly communicate policies to participants and get confirmation that they understand and can abide by your policies.
Plan policies for scenarios that might arise in an environment that is not managed by the test program administrator or proctor. For example, are you legally allowed to video someone who passes by in the background if they have not given their permission to be recorded? If not, have a policy in place stating that the participant is responsible for finding an isolated place to test. Do you or the proctoring company manage the location where the test is being delivered? If not, have a policy for who takes responsibility and absorbs the cost of an unexpected interruption like a fire alarm or power outage.
You should be prepared to document the comparability of administrations. This might include describing potential variations in the remote environment and how they may or may not impact the assessment results and security.
It is also advisable to audit some administrations to make sure that the testing environments comply with your testing program’s security policy. The online proctors’ incident reports should also be recorded in an administration report, just as they would with an in-person proctor.
You also need to make sure that everything needed to administer the test is provided, either physically or virtually.
Each participant must have the equipment and resources needed to take the test. If it is not reasonable to expect the participant to handle these tasks, you need to plan for someone else to do so, just as you would at a test center. For example, it might not be reasonable to expect some participant populations to know how to check whether the computer used for testing meets minimum software requirements.
If certain hardware (e.g., secured computers, cameras, scanners, microphones) or test materials (e.g., authorized references, scratch paper) are needed for the assessment design, you need to make sure these are available onsite for the participant and make sure they are collected afterwards.
Accommodations may take the form of physical or virtual test materials, but accommodations can also include additional services or some changes in the format of the assessment.
Some accommodations (e.g., extra time, large print) can be controlled by the assessment instrument or an online proctor, just as they would in a test center.
Other accommodations require special equipment or personnel onsite. Some personnel (e.g., scribes) may be able to provide their services remotely, but accommodations like tactile printouts of figures for the blind must be present onsite.
Extra effort is clearly needed when setting up an online-proctored test. Activities that might have been handled by a testing center (control of the environment, management of test materials, providing accommodations) now need to be remotely coordinated by the test program staff and proctors; however, the payoffs may be worth the extra effort. If comparable administration practices can be achieved, online-proctored assessments may be cheaper than test centers, offer increased access to participants, and lower the risks of collaborative cheating.
As we bid 2014 goodbye and welcome 2015, we wish all readers of this blog a happy and prosperous new year.
Trust is in the news a lot these days. 2013 was memorable for its revelations of government surveillance of the Internet. Well-intentioned government organizations were intercepting Internet communications for law and order purposes, and to protect society from harm. However the surprising scale of the interceptions divided the community with some feeling it was appropriate given the threat but others becoming less trustful of government.
In 2014, we have seen a series of Internet vulnerabilities. The catchy names – Heartbleed, Shellshock and Poodle – bely the potential seriousness of these threats. Questionmark was only lightly touched by these vulnerabilities and any minor issues were quickly corrected (see Questionmark and Heartbleed and Questionmark not impacted by Bash/ShellShock Internet vulnerability). However as we’ve seen in the news, some other companies have been impacted by these or other vulnerabilities, and we are all very sensibly being more cautious about security and data protection.
2014 seems to have been the year that security and data protection have come of age. Mature organizations recognize that there are significant security threats to their data, and mature suppliers put in place extensive measures to protect against such threats. The arguments in favour of outsourcing to the Cloud remain strong; if nothing else, Cloud providers can typically protect specialist data like assessments better than a busy in-company IT team, who are focused elsewhere. But trust must be at the forefront – you need to trust and review all your suppliers, to check that they are following good security practice. We welcome all the review we get from our customers’ IT security departments – good questions help make us stronger.
Trust and trustable assessment results are critical to Questionmark. Our vision is that in today’s world, success for organizations, individuals and society means having the right knowledge, skills and abilities at the right place and the right time. An organization needs to know what its people understand and what they need to change or learn to meet goals. An individual needs to demonstrate achievement and find out how to improve. And society needs to know who is competent and whom to trust.
Assessments are critically needed to identify if people “know it, understand it and can do it”. Questionmark aims to provide the world’s leading online assessment service, allowing organizations to securely create, deliver and report on tests, quizzes, surveys and exams. Questionmark focuses on getting trustable results that are actionable for organizations, individuals and society.
During 2015 we’ll be sharing lots about assessment and good practice on this blog, and I trust we will have much to interest you!
Although you can use Questionmark as a stand-alone Assessment Management System (AMS), it also integrates seamlessly with other key systems – everything from learning management systems and content management systems to portals and scanning technologies.
We also support integrations with LTI-, AICC– and SCORM-compliant systems.
You can find video tutorials about many of these connectors in the Questionmark web site. There you’ll find videos on integrating with Moodle, SuccessFactors and Cornerstone OnDemand, as well as other systems such as SharePoint, Canvas, and Ning.
The Questionmark Blackboard Connector is a proprietary connector that provides unprecedented integration between the Blackboard LMS and Questionmark. Through the Blackboard Connector:
The first time an instructor interfaces with Questionmark, a Questionmark admin ID is created for them automatically.
When an instructor adds a Questionmark assessment to a Blackboard course, the course short name becomes a Questionmark group, and the instructor and any
students launching the assessment are automatically added to the group.
The first time a student launches any Questionmark assessment, a participant ID is created in Questionmark for the student.
And all of this automatic synchronization is optional! You can just as easily set up the connector to require that instructors, students and/or groups be created by a Questionmark admin in Questionmark so that you can control exactly who, and what courses, can interface with Questionmark.
Watch this video for a complete explanation of what the Blackboard Connector can do for you: