What is the Single Best Way to Improve Assessment Security?

John KleemanPosted by John Kleeman

Three intersecting circles, one showing Confidentiality, one showing Availability and one showing IntegrityAssessment results matter. Society relies on certifications and qualifications granted to those who pass exams. Organizations take important decisions about people based on test scores. And individuals work hard to learn skills and knowledge they can demonstrate in tests and exams. But in order to be able to trust assessment results, the assessment process needs to be secure.

Security is usefully broken down into three aspects: confidentiality, integrity and availability.

  • Confidentiality for assessments includes that questions are kept secure and that results are available only to those who should see them.
  • Integrity for assessments includes that that the process is fair and robust, that identify of the test-taker is confirmed and that cheating does not take place.
  • Availability includes that assessments can be taken when needed and that results are stored safely for the long term.

A failure of security, particularly one of confidentiality or integrity reduces the usefulness and trustworthiness of test results. A confidentiality failure might mean that results are meaningless as some test-takers knew questions in advance. An integrity failure means that some results might not be genuine.

So how do you approach making an assessment program secure? The best way to think about this is in terms of risk. Risk assessment is at the heart of all successful security systems and central to the widely respected ISO 27001 and NIST 800-53 security standards. In order to focus resources to make an assessment program secure and to reduce cheating, you need to enumerate and quantify the risks and identify probability (how likely they are to happen) and impact (how serious it is if they do). You then allocate mitigation effort at the ones with higher probability and impact. This is shown illustratively in the diagram – the most important risks to deal with are those that have high probability and high impact.

Four quadrants showing high probability, high impact in red and Low probability, low impact in green. With yellow squares for high probability, low impact and low probability, high impact

One reason why risk assessment is sensible is that it focuses effort on issues that matter. For example, the respected Verizon data breach investigations report for 2017 reported that 81% of hacking-related breaches involved weak or stolen passwords. For most assessment programs, it will make sense to put in place measures like strong passwords and training on good password practice for assessment administrators and authors to help mitigate this risk.

There is no “one size fits all approach”. Some risks will differ between assessment programs. To give a simple example, some organizations are concerned  about people having reference materials or “cheat sheets” to look up answers in and this can be an important risk to mitigate against; whereas in other programs, exams are open book and this is not a concern. In some programs, identity fraud (where someone pretends to be someone else to take the exam for them) is a big concern; in others the nature of the proctoring or the community makes this much less likely.

If you’re interested in learning more about the risk approach to assessment security, I’m presenting a webinar “9 Risks to Test Security (and what to do about them)” on 28th November which:

  • Explains the risk approach to assessment security.
  • Details nine key risks to assessment security from authoring through delivery and into reporting.
  • Gives some real examples of the threats for each risk.
  • Suggests some mitigations and measures to consider to improve security.

You can see more details on the webinar and register here.

Assessment security matters because it impacts the quality and trustworthiness of assessment results. If you are not already doing it, starting a risk-based approach to analyzing risks to your security is the single best way to improve assessment security.

Exams and social media: is it really spying?

Steve Lay HeadshotPosted by Steve Lay

While I was traveling back from our US Users Conference several weeks ago, a debate was raging on social media following news that a testing company had been monitoring Twitter to detect evidence of leaked content. The Guardian newspaper, for example, reported that a New Jersey superintendent had found this ‘disturbing’.

In case you haven’t read about this case, here are the basics: after school, a student tweeted information about a test administered earlier that day. An automated Web monitoring system discovered the tweet, and the school was notified. The student later deleted the offending tweet.

According to the test provider, administrators are supposed to tell participants that sharing any test question online is prohibited. It isn’t clear from the press reports whether this warning was issued prior to the test or whether the student would have considered the tweet prohibited or not. Whatever the case may be, enough information was shared to trigger the automated warning.

Perhaps more interesting than the story itself is the reaction to it. Strong words have been used, but should monitoring social media really be regarded as spying?

The monitoring of online forums to check for exam leaks is not new. It goes back to the very earliest days of the Internet. When I first read about this case my first reaction was that this type of thing is happening all the time. Indeed, brand owners are constantly monitoring social media to help them understand the public’s reaction to their products and services and to help them target their advertising more effectively. Copyright owners also monitor the web to check for infringement. Trademark owners must pro-actively monitor for misuse to prevent their trademarks from becoming unenforceable. So if an organization has such rights, wouldn’t monitoring the web–including social media–to enforce them surely be expected?

This assumption is probably naive. Many people are not aware that this information is available in a form that can be subscribed to. They do not understand the subtle difference between a comment being made in a ‘public place’ like twitter and it being instantly discoverable. In our everyday experience, a conversation that happens in a public place like a café or store is not recorded, transcribed and then made instantly available to business partners of the venue. In this case, the student, the student’s parents and even the superintendent were surprised and shocked by the level of surveillance. They reacted as if a private conversation had been overheard.

It is interesting to contrast this recent case with one reported by Techcrunch in 2009, when information from Facebook was used to hold students to account for cheating. But in the Facebook case, the information was discovered by other students and brought to the attention of the test authorities. Why would the students do that? Likely because test takers are key stakeholders too! If cheating becomes commonplace, then the test will become worthless. So both the test publisher and the test taker have an interest in ensuring fair practice.

Coming back to the rogue tweet, what’s frustrating here is that there is no suggestion that the test taker was trying to cheat or to help someone else cheat. I haven’t seen the 140 characters in question, but it seems likely that the tweet was just a trivial extension of the type of verbal conversation that people frequently have after taking tests.

The mismatch in privacy expectations and the feeling that the student was being accused of malpractice were a toxic mix. Both of these can be avoided.

When monitoring people using CCTV or similar technologies, it is good practice to inform people that they are being monitored, and for what purpose. In many jurisdictions this may also be a legal requirement. Likewise, why not inform test takers of the type of monitoring that is taking place and why? This may have the added advantage of helping to inform them about the risks to their own privacy that over-sharing on social media can pose.

Also, when issues are flagged by monitoring services, test publishers should think carefully about any follow-up actions. Are these actions consistent with the stated purpose of the monitoring? Are they proportionate? Remember, the test taker and the test publisher should be on the same side!

What is the best way to reduce cheating?

John Kleeman HeadshotPosted by John Kleeman

There is a famous saying: “If you want to build a ship, don’t drum up the people to gather wood, divide the work, and give orders. Instead, teach them to yearn for the vast and endless sea.” This has a useful analogy in preventing cheating.

There are many useful technical and procedural ways of preventing cheating in tests and exams, and these are important to follow, but an additional, cost-effective way of reducing cheating is to encourage participants to choose not to cheat. If you can make your participants want to take the test fairly and honestly — by reducing their rationalization to cheat — this will reduce cheating.

Fraud triangle - Motivation, Opportunity and RationalizationAs shared by my colleague Eric Shepherd  in his excellent blog article Assessment Security and How To Reduce Fraud, cheating at a test is a variant of fraud.  Donald Cressey, a famed criminologist came up with the fraud triangle shown in the diagram to the right to explain why people commit fraud.

In order for someone to commit fraud (e.g. cheat at a test), he or she must have Motivation, Opportunity and Rationalization.  Motivation comes from the stakes of the test; for an important test, this is difficult to reduce. Opportunity arises out of technical and procedural weaknesses in the test-taking process, and you can obviously strengthen processes to reduce opportunity in many ways.

Rationalization is when someone reconciles their bad deeds as acceptable behavior. We all have values and like to think that what we are doing is right. When someone conducts fraud, they typically rationalize to themselves that what they are doing is right or at least acceptable. For example, they convince themselves that the organization they are robbing deserves it or can afford the loss. When cheating at a test, they say to themselves that the test is not fair or that they are just copying everyone else or they find some other excuse to rationalize and feel good about the cheating.

Here are some ways to make it less likely that people will rationalize about cheating on your test:

1. Formalize a code of conduct (e.g. honesty code) which sets out what you expect from test takers. Communicate this effectively well in advance and get people to sign up to it right before taking the test. For example, you can put it on the first screen after they log in. This will reduce rationalization from people who might claim to themselves they didn’t know it was wrong to cheat or that everyone cheats.

2. Provide fair and accessible learning environments where people can learn to pass the assessment honestly, and provide practice exams so people can check their learning. Rationalization is increased if people think there is no other way to pass the test than cheating.

3. Make sure that the test is trustable (reliable and valid) and fair. If the test is not seen as fair,  people will be less like to rationalize that it’s permissible to cheat.

3. Communicate details of why the tests are there, how the questions are constructed and what measures you take to make the Cheat sheet in a juice box test fair, valid and reliable. Again, if people know the test is there for good reason and fair, they will be less motivated to cheat.

4. Maintain a positive public image. This will reduce rationalization by people claiming that  the assessment provider is incompetent or has other faults.

5. Communicate your security measures and how people who cheat are caught.  This makes people less likely to think they will be able to get away with it.

For many organizations — in addition to other anti-cheating measures — it can be very productive to spend time reducing participants’ rationalization to cheat, thereby helping them choose to be honest. The picture on the right shows a “cheat sheet” or “crib sheet” hidden in a juice carton. Think of ways you can encourage participants to use their inventiveness to learn to pass the exam, not to believe it’s okay to defraud you and the testing system.

I hope you find this good practice tip helpful. I’ll be presenting at the Questionmark Users Conference March 10 – 13 on Twenty Testing Tips: Good practice in using assessments. Taking measures to reduce rationalization for cheating will be one of my tips. Register for the conference if you’re interested in hearing more.

9 Tips to Prevent Cheating and Ensure Test Security

Chloe MendoncaPosted by Chloe Mendonca

The security of test results is crucial to the validity of test scores. Check out 9 tips to prevent cheating and ensure test security in the  infographic below.

If you’d like more details about these and other tips on ensuring the security and defensibility of your assessments you can download our white paper: Delivering Assessments Safely and Securely. [Free after registration]

Prevent Cheating and Ensure Test Security from Questionmark‘s Slideshare page

Secure Testing in Remote Environments: A SlideShare Presentation

Headshot JuliePosted by Julie Delazyn

How can you be sure that someone taking an online exam away from a testing center or classroom is adhering to the guidelines put in place by your instructional staff?

This SlideShare presentation will demonstrate how instructors can prevent or catch cheating and ensure a secure environment for employees or students taking tests in their homes, offices and other locations.

The slides are from a Best Practices sessions at the 2013 Questionmark Users Conference: Don Kassner of ProctorU discussed strategies for reducing incidents of dishonesty online, and Maureen Woodruff of Thomas Edison State College explained how online proctoring enables the college to administer tests securely to thousands of online learners.

This presentation offers a glimpse into the kind of discussions and sessions you can find at our Users Conferences. Registration is already open for the 2014 Users Conference March 4 – 7 at the Grand Hyatt on the beautiful Riverwalk in San Antonio, Texas. Discounts are available for groups and early registrants. Sign up soon and plan to be there!

Secure exams outside the testing center

Joan Phaup HeadshotPosted by Joan Phaup

The increasing numbers of students studying online in recent years – many of them raising families and holding down jobs – have embraced the idea of doing all their coursework at the kitchen table, so to speak.  But until recently, when it came time for a test, these students had to travel to at testing center. Many of these students raised the question: “If I can study at the kitchen table, why can’t I take an exam there, too?”

Today, taking tests from home or the office — using online monitors or proctors — is an option for certification candidates as well as students, and there are various means of providing secure testing at a distance.

Delegates to the Questionmark Users Conference in Baltimore March 3 – 6 will have the opportunity learn more about online proctoring/invigilation during a presentation on Secure Testing in Remote Environments.

Don Kassner, president of ProctorU, will co-present this session with Maureen Woodruff, who directs the Office of Test Administration at Thomas Edison State College. I spent a few minutes with Don the other day and asked for some details.

Can you explain what makes it possible to offer secure remote proctoring or monitoring?

Don Kassner

Don Kassner

There are three key elements to this: the environment, the computer and the test taker. The first thing we need to do is to make sure each test taker has reliable internet access and is in a fairly controlled environment. This is not about testing anywhere. It’s about testing in an environment that’s predictable. The person gets to choose the place, but it has to be in a certain kind of place. And the test taker must “show” us their environment using a webcam. Second, we have to secure their computer. Test takers use their own equipment, but we need to make sure they are not switching tasks, accessing the Internet for answers and so forth. Last, we have to secure the test taker themselves, by using a layered authentication approach to make sure they are who they say they are having our online proctors observe them as they complete their tests.

What are the biggest security challenges in delivering tests to people outside of test centers?

In a test center, you already know that the environment and the computers are secure; you can focus on the identity and behavior of the test taker.  When proctoring at a distance, you have to give a lot of importance to all three of the elements I mention; there are a lot more decisions to be made about the testing process.

With online proctoring, we have to be willing to stop a testing session and say that something doesn’t meet our standards – that the test taker is not meeting the requirements and must reschedule.

We also need to be able replicate our processes across the board and make sure that the testing experience replicates no matter who is taking the test or where they are taking it. We have to focus on making sure the experience is identical for every test taker.

How will you address those challenges during your session?

We will introduce the basics of online proctoring and give examples of how different institutions and organizations have used it. We’ll also drill down into the details of what it takes to secure the environment, the computer and the test taker.

Woodruff Maureen Exc  Portrait (cropped)

Maureen Woodruff

Maureen will share a case study about what they did at Thomas Edison State College and the important factors they had to take into account when they set up remote testing for their students. And I’ll differentiate between the factors that are important for academic institutions and those that matter the most for certification tests. Students are likely to take a number of tests and end up have a long track record. Certification candidates tend not to be repeat test takers, so that means using slightly different procedures.

What kinds of tests are best suited for online monitoring or proctoring?

If you are going to use this kind of proctoring, you really need to think about the nature and structure of the test. You are trying to minimize the risk inherent in someone taking a test, so you need to ask yourself what issues you are concerned about relative to that. Tests with large data banks are best, because they help mitigate the risks of people stealing questions or colluding. Standard tests  increase the risk factor and may not be appropriate.

What would you like your audience to take away?

A real understanding on how effective this approach can be in some situations and  an understanding of when it may or may not be appropriate – so they can think about their own programs and consider where they think this will fit.

Click here for more information about the conference program — and register soon!