2014 South African Users Conference – Addressing Compliance

Austin FosseyPosted by Austin Fossey

We are back from the first South African Users Conference which was hosted by Bytes People Solutions. Like all of our users conferences, the most valuable aspect of this gathering was hearing from our customers and potential customers—through presentations as well as informal conversations.

Many attendees manage assessment programs for large academic or commercial institutions, and I was struck by their teams’ organizational skills. From my conversations, it sounds as if many of these program managers have to strike a balance between traditional practices at their organizations and the needs to adopt innovative strategies to improve measurement practices. For example, one program manager spoke about helping item writers transition from writing items in MS Word to writing them in Questionmark Live. The people I spoke to appeared to be pushing the envelope of their assessment capabilities, helping their stakeholders through technological transitions, while simultaneously delivering thousands of assessments. It was impressive.

Compliance was a recurring theme. In the U.S., test developers are always collecting evidence to demonstrate the legal defensibility of their assessments, and we often turn to The Standards for Educational and Psychological Testing for guidance (the latest edition was released just last week). Though the legal and cultural expectations for test development may differ slightly in other regions, no modern test developer is exempt from accountability. Demonstrating compliance with organizational or legal requirements seemed to be a big consideration for many attendees.

Regardless of what compliance means to different organizations, one thing was the same for everyone: demonstrating compliance means having accurate, easily-accessed data. I noticed that many clients were able to cite data-backed evidence for the decisions they made in their testing programs to meet their stakeholders’ compliance requirements. Some of these data came from Questionmark through our APIs and assessment results, but these presenters also clearly did research about other important factors that impact the validity of the results.

For example, presenters talked about the evidence they gathered to support the use of computer-based testing over paper and pencil tests. Another presenter shared qualitative data from interviewing subject matter experts about their impressions of Questionmark’s authoring tools. These decisions affect the delivery mode and task models of the assessment, which directly relate to the validity of the results, so it is encouraging to see test developers documenting their rationales for these kinds of decisions.

All in all, it was an impressive group of professionals who gathered in Midrand, and I am sure that I learned just as much (if not more) from the participants as they did from me. Special thanks to everyone who attended and presented!

Defense in Depth: Security for SCORM and Beyond


Posted by Tom King

My earlier post, The Importance of Security and Integrity of Performance Data addressed a specific emerging SCORM security issue. It also raised the issue of “Defense in Depth” as an approach for improving security. Here are some defense in depth approaches you can use right now to increase security and decrease vulnerability.

Key ways to reduce vulnerability and improve security.

  • Audit trails and accountability. Have a second source of data to cross-check. Ideally this data should be automatically collected. Data sent to a SCORM or AICC LMS is also sent to a Questionmark Perception server via a different data conduit.
  • Secured Communication. Transfer responsibility for the result data to a server. Questionmark’s secure server-to-server implementation of AICC does this.
  • Increased Client/Browser Security. Reduce the attack surface of the runtime. Use a Secured Browser that disables or limits functionality not directly needed for the primary activity. Questionmark Secure is a browser that does this for AICC or SCORM.
  • Direct Proprietary Communication. This approach works by centralizing the chain-of-custody for the data to one trusted provider. Questionmark Perception can manage the process completely from authoring to scheduling to delivery to reporting.

Audit trails. Keeping parallel records such as with a double-entry accounting system is one way to achieve an audit trail. Having such an audit trail is key to identifying and recovering from errors or misdeeds. Questionmark provides capabilities for such an audit trail through both its SCORM and its AICC implementations. Perception achieves increased security and this audit trail by sending data to the LMS using the SCORM or AICC standard and, in parallel, sending data directly to the secure Perception server database. In the case of an error or misdeed, the LMS system results and the results in the secured Perception database can be compared to recover from either a security breach or an error.

Secured Server-to-Server Communication. In the cheatlet exploit, the openness of the published SCORM API and the browser JavaScript layer are used to inject false data from the client side. One way to increase the security is to remove this client side vulnerability and use AICC instead of SCORM. The innovative Perception server-to-server implementation of the AICC HACP specification demonstrates this, by having the browser relay minimal data to the Perception server. The client is not capable of directly injecting falsified overall score data. The Perception server is ultimately responsible for judging response and data communication with the LMS, not the browser client.

In 2002, Paul Roberts of Questionmark identified and described the risks of the client-side API (see Security Issues with the JavaScript API, Paul Roberts, 2002 on the AICC web site). He urged the AICC to continue to support the HACP protocol because of the value of the increased security enabled with a server-to-server AICC implementation. The diagram below helps explain this communication.


Increased Browser Security. As currently implemented, this exploit relies on user access to the UI to open a bookmark. Changes to the launch environment (browser) can reduce this vulnerability. The Questionmark Perception Secure Browser is a commercialized browser solution built for the rigorous requirements of high-stakes testing environments. When a participant takes an online assessment using Questionmark Secure, the secure browser displays the HTML content of the assessment, but disables key functions such as task-switching, right click options, screen captures, menus and printing. There simply isn’t a means to access a menu or bookmark to trigger.

Direct Proprietary Communication In this scenario, one trusted party is responsible for the full span of access, delivery, and results. It does run somewhat contrary to cybersecurity practice of published protocols and specifications that can bear wide scrutiny. It can also undermine interoperability, something near and dear to my heart. In the long run, I believe you’ll find Questionmark moving in directions that addresses these type of concerns.

However, there are many valid circumstances where the values of single party chain of custody and trusted relationship trumps other concerns. High stakes test are often the prime case for this, and it is critical to expand cyber-defense-in-depth with adjunct security measures (such as tight control of source materials, exam monitors, proctors/invigilators).

Work-around versus defend-against. Finally, as an exercise for the reader, you may consider reading the the two ADL workarounds published April 2, 2009. You’ll find that the excerpt on Securing Your Assessments provides a means of masking the location of answer-judging source code sent to the client by some systems. While useful, it doesn’t provide the same security and depth of defense as other approaches. Consider for instance using Questionmark Secure (prevents ‘view source’) with the Perception SCORM implementation (adds audit trail) and Perception server-side evaluation logic (secures the evaluation logic on the server-side). That is defense in depth. One might even replace SCORM with AICC in this case for additional security in addition to or in lieu of Questionmark Secure.

Whenever faced with security concerns regarding the possibility of cheating, abuse or data integrity, I encouraged you to think about defense in depth and the role of all the components in security.