My previous post offered four tips on making your assessments more secure and preventing cheating.  Aside from verifying IP addresses and running a Trojan horse or stealth items to help detect whether a participant has memorized the answer key, there are some physical actions you can take to avoid the problem and reduce the temptation to cheat.

Proper seating arrangements for participants

Seating participants with adequate space between them and giving them limited ability to see another participant‘s screen or paper are important strategies for enhancing test security. The proctor should be aware of cheating techniques such as the ―”flying V” seating arrangement where the “giver” at the point of the V feeds information to a number of “receivers” behind them. The givers and receivers can communicate in a number of ways, using sign language, dropping notes on the floor, etc. (Dr. Gregory Cizek’s book “Cheating on Tests: How to Do it, Detect it, and Prevent it,” will tell you  more about this and other aspects of cheating.)

Example of the “flying V” answer copying formation (Cizek, 1999):

Using unique make-up exams

Many organizations offer make-up exams for participants who were sick or had legitimate excuses for not being able to take an assessment at the scheduled date and time. If you use the same exam that was administered at the scheduled date and time for their make-up exam, you open yourself to risks of the exam form being compromised. Sometimes the make-up exams are not administered in the same strict proctored environment as the scheduled exam, allowing participants the opportunity to cheat or steal content.

Using more constructed response questions

Constructed response questions, like essay or short answer questions, provide less opportunity for participants to cheat because they require them to produce unique answers to questions. There is no answer key to steal, and participants who copied other people’s constructed response answers are easily identified via a side-by-side comparison of answers.

I hope you enjoyed this three part series on preventing cheating.  You will find more information about  various means for deploying many different types of assessments in our white paper, “Delivering  Assessments Safely and Securely.”

My previous post offered three tips on making your assessments more secure and preventing cheating. Here are four more. You will find additional information about this in “Delivering  Assessments Safely and Securely,” and I’ll be mentioning other security tips in my future posts. I hope you will respond with your own ideas about avoiding the problem of cheating on tests.

Screening participants who achieve perfect scores

Given the rarity of achieving perfect scores on assessments, consider doing some investigating when you see perfect scores. Many organizations do this automatically. This might  interview the exam proctor and do other checks to ensure no suspicious behavior has occurred.

Verifying expected IP addresses

If you are administering an assessment  at a specific location, you will likely be able to obtain  the IP address of the computer being used. You can then tell whether participants took the assessment there or at an unauthorized location.

Using Trojan horse or stealth items

Use Trojan horse or stealth items to help detect whether a participant has memorized the answer key. Stealth items look just like the other questions, but they are purposely keyed incorrectly. You can include these items are generally included as non-scored items on the assessment. They will help you detect if a participant is simply memorizing content and keyed correct answers, since they will likely choose alternatives that they have memorized. Participants with overall reasonable assessment scores who got the stealth items “correct” might have memorized the answer key.

Reveal that cheater prevention tactics are used

Informing participants that reviews are regularly conducted to identify cheaters is a simple way to decrease the temptation to cheat. You don’t need to provide details about the sort of reviews you conduct, but do let participants know that cheater-detection tactics are regularly employed.

Protecting the intellectual property contained in a test or exam is essential, not only because of the time, effort and cost of creating assessments but also because IP theft undermines the accurate measurement of knowledge and skills.

Protecting intellectual property protects the credibility of tests. Here are four tips for helping to ensure the security of intellectual property:

Create and administer multiple test forms

Rather than having only one form of the assessment being administered, delivering multiple forms of the same exam can help limit item exposure. This method also allows for the possibility of interspersing large-scale integrated beta test questions within the forms to collect psychometric information on newly developed questions.

Restrict and control administration of beta test items

Beta testing questions is an important part of high-stakes assessment, ensuring the psychometric quality of questions before they appear on actual assessments. However, it is vital that a well conceptualized beta test model is in effect to limit the exposure of newly developed questions to participants.

Update exam forms periodically

Letting exam forms become stale can over-expose questions to participants, increasing the likelihood of IP theft. An organization could consider retiring old exam forms and turning them into exam prep materials that can be sold to participants. In this way, participants could periodically expect new practice questions.

Produce exam prep materials

Organizations should consider making exam prep materials available to participants before an assessment. This will help reduce the demand for participants to try to obtain exam questions via illegal means as they will have access to the type of questions that will be asked on the actual assessment.

For more details on this subject, plust information about various means for deploying a wide range of assessment types with assurance, download our White Paper: Delivering Assessments Safely and Securely.

Today I’d like to share this chart that can help you determine what delivery environment would be suitable for various types of assessments. The chart appears in our white paper, Delivering Assessments Safely and Securely, which offers many more details on this subject–the aim being to help you avoid incurring unnecessary costs while at the same time giving your assessments the security levels they require.

Posted by Joan Phaup

I enjoyed talking recently with Shannon Bonner of Southern California Edison about the importance of good test maintenance and how to establish solid test maintenance practices. Listen to this podcast for tips about how to maintain the underlying validity of  assessments, ensure the quality of questions and protect test security.

Posted by Greg Pope

At last week’s Questionmark Users Conference I presented a session on item and test analysis, and part of that session dealt with test score reliability.

“Reliability” is used in everyday language: “My car runs reliably” means it starts every time. In the assessment realm we talk about test score reliability, which refers to how consistently and accurately test scores measure a construct (knowledge/skills in the domain of interest such as “American History Knowledge”).

Assessments are measurement instruments; the questions composing the assessment take measurements of what people know and can do. Just as thermometers take measurements of temperature, assessment questions take measurements of psychological attributes. Like any measurement instrument, there is some imprecision in the estimates, so the test score that a person obtains (observed score) is actually composed of a theoretical “true score” (what they actually really know and can do) plus some error. Reliable test scores have the least amount of error and therefore the smallest difference between the observed score and this theoretical true score.It is hard to go into a great deal of detail here, so for a good primer into the theory check out: Traub, R.E. (1994). Reliability for the Social Sciences: Theory & Applications. Thousand Oaks: Sage.

Generally there are four approaches for measuring reliability:

1.    Internal consistency: How well do items on the test “hang together” to measure the same psychological attribute

2.    Split-half (split-forms): How well do scores on two forms (splits) of the test (first 25 items versus last 25) relate to one another

3.    Test-retest: How similar are scores obtained from multiple administrations of the same test

4.    Inter-rater reliability: How consistently do two or more raters (essay markers) obtain similar scores.

Internal consistency reliability is common and is used in our Test Analysis Report and Results Management System ,where we use Cronbach’s Alpha.

Stay tuned for Part 2 of this post, which will discuss the factors and test characteristics that generally influence internal consistency reliability coefficient values.

Posted by Tom King

My earlier post, The Importance of Security and Integrity of Performance Data addressed a specific emerging SCORM security issue. It also raised the issue of “Defense in Depth” as an approach for improving security. Here are some defense in depth approaches you can use right now to increase security and decrease vulnerability.

Key ways to reduce vulnerability and improve security.

• Audit trails and accountability. Have a second source of data to cross-check. Ideally this data should be automatically collected. Data sent to a SCORM or AICC LMS is also sent to a Questionmark Perception server via a different data conduit.
• Secured Communication. Transfer responsibility for the result data to a server. Questionmark’s secure server-to-server implementation of AICC does this.
• Increased Client/Browser Security. Reduce the attack surface of the runtime. Use a Secured Browser that disables or limits functionality not directly needed for the primary activity. Questionmark Secure is a browser that does this for AICC or SCORM.
• Direct Proprietary Communication. This approach works by centralizing the chain-of-custody for the data to one trusted provider. Questionmark Perception can manage the process completely from authoring to scheduling to delivery to reporting.

Audit trails. Keeping parallel records such as with a double-entry accounting system is one way to achieve an audit trail. Having such an audit trail is key to identifying and recovering from errors or misdeeds. Questionmark provides capabilities for such an audit trail through both its SCORM and its AICC implementations. Perception achieves increased security and this audit trail by sending data to the LMS using the SCORM or AICC standard and, in parallel, sending data directly to the secure Perception server database. In the case of an error or misdeed, the LMS system results and the results in the secured Perception database can be compared to recover from either a security breach or an error.

Secured Server-to-Server Communication. In the cheatlet exploit, the openness of the published SCORM API and the browser JavaScript layer are used to inject false data from the client side. One way to increase the security is to remove this client side vulnerability and use AICC instead of SCORM. The innovative Perception server-to-server implementation of the AICC HACP specification demonstrates this, by having the browser relay minimal data to the Perception server. The client is not capable of directly injecting falsified overall score data. The Perception server is ultimately responsible for judging response and data communication with the LMS, not the browser client.

In 2002, Paul Roberts of Questionmark identified and described the risks of the client-side API (see Security Issues with the JavaScript API, Paul Roberts, 2002 on the AICC web site). He urged the AICC to continue to support the HACP protocol because of the value of the increased security enabled with a server-to-server AICC implementation. The diagram below helps explain this communication.

Increased Browser Security. As currently implemented, this exploit relies on user access to the UI to open a bookmark. Changes to the launch environment (browser) can reduce this vulnerability. The Questionmark Perception Secure Browser is a commercialized browser solution built for the rigorous requirements of high-stakes testing environments. When a participant takes an online assessment using Questionmark Secure, the secure browser displays the HTML content of the assessment, but disables key functions such as task-switching, right click options, screen captures, menus and printing. There simply isn’t a means to access a menu or bookmark to trigger.

Direct Proprietary Communication In this scenario, one trusted party is responsible for the full span of access, delivery, and results. It does run somewhat contrary to cybersecurity practice of published protocols and specifications that can bear wide scrutiny. It can also undermine interoperability, something near and dear to my heart. In the long run, I believe you’ll find Questionmark moving in directions that addresses these type of concerns.

However, there are many valid circumstances where the values of single party chain of custody and trusted relationship trumps other concerns. High stakes test are often the prime case for this, and it is critical to expand cyber-defense-in-depth with adjunct security measures (such as tight control of source materials, exam monitors, proctors/invigilators).

Work-around versus defend-against. Finally, as an exercise for the reader, you may consider reading the the two ADL workarounds published April 2, 2009. You’ll find that the excerpt on Securing Your Assessments provides a means of masking the location of answer-judging source code sent to the client by some systems. While useful, it doesn’t provide the same security and depth of defense as other approaches. Consider for instance using Questionmark Secure (prevents ‘view source’) with the Perception SCORM implementation (adds audit trail) and Perception server-side evaluation logic (secures the evaluation logic on the server-side). That is defense in depth. One might even replace SCORM with AICC in this case for additional security in addition to or in lieu of Questionmark Secure.

Whenever faced with security concerns regarding the possibility of cheating, abuse or data integrity, I encouraged you to think about defense in depth and the role of all the components in security.

Posted by Tom King

There’s been much discussion among developers and insiders about recent posts and vendor notices regarding automated cheating tools for SCORM 1.2/SCORM 2004 content. I’d like to share some thoughts on the underlying issues and measures that can be taken to reduce vulnerability. This is no joke… unlike the recent fun on Eric’s blog.  This is the first in a series of posts on this topic. I’d like to review a few key concepts before delving into specific areas and solutions that will come in the posts to follow.

Security is important. The cornerstone of successful education, training and certification is the effective use of assessments. Inaccurate, misleading or falsified performance data can lead to poor decisions, increased liability and other significant risks or consequences. Questionmark is keenly aware of the importance of security and has addressed the issue from the beginning.

Vulnerabilities exist. Note that this is vulnerabilities in the plural. About a week ago, an independent elearning developer published one: a simple bookmarklet to send false score and status data to a SCORM LMS (see Cheating in SCORM, Phillip Hutchison, 2009). [Note: The post remains, but the sample bookmark has been removed after an outside party requested this.]

Using the published exploit is as simple as saving a book mark to your browser and then picking that bookmark while content is running. It is what I’ve called a “cheatlet” and current implementations may foreshadow other potential issues (see Security Before Features, Tom King, 2008 on LETSI SCORM 2.0 web site). Others have discussed how common in-browser debugging tools like Firebug can be used to similar effect. The key message is that this type of exploit is possible, and it gets easier and more viable over time.

Defense in depth. That phrase is a bit of a mantra in the cybersecurity world. Questionmark has taken this approach with its implementation of interoperability solutions, including SCORM. Some in the SCORM community recognize the “cheatlet” exploit as a known weakness that has just become easier for the common man to use. They go on to indicate that SCORM shouldn’t be used for high stakes assessments, and end their argument or response there. It is left for future specifications to deal with this issue. However there are several alternatives to decrease vulnerability.

I think I now have your attention for the subsequent posts. Also, if you’re attending the Questionmark User Conference 2009 this coming week in Memphis, please feel free to  stop by for my session on standards and ask about this or other standards-related issues.

Posted by Julie Chazyn

We have just updated our white paper on Delivering Assessments Safely and Securely, which helps people choose security measures that match up with the types of assessments they’re delivering – from low stakes to high stakes.

This new paper takes into account changes in technologies and standards that have taken place in the last few years—as well as new testing environments and methods. We’ve also added some tips to help prevent cheating.

You can download the paper here.