Posted by Joan Phaup

Steve Lay

Daniel R. Rehak, a technical advisor to the Advanced Distributed Learning Initiative (ADL), is responsible for technical and strategic advice primarily about the direction and refinement of ADL’s activities in the development of SCORM and content repositories and registries. As part of its plan to release an enhanced, harmonized version of its SCORM interoperability standards in 2011, ADL is actively seeking the views of SCORM and AICC community members. Dan is coming to the Questionmark Users Conference, takinig place in Miami March 14 – 17,  to hear what customers hope to see in an updated version of SCORM. He and Questionmark Integration Team Lead Steve Lay will co-facilitate a discussion on Shaping the Future of ADL SCORM:  What’s On Your Wish List?

I recently asked Dan about his ADL work and his desire to sit down with Questionmark users to discuss the future of SCORM:

Q: How would you define SCORM?

Daniel Rehak

A: SCORM is a model — a collection of different interoperability standards that for the most part were originally created to work on their own. SCORM explains how to make them work together with the objective of taking learning activities, managed by a learning management system, and making them portable, interoperable and reusable across different vendors and technology platforms. In addition to being used by many commercial organizations and the military, it’s used for K12 education in the UK and Korea. It’s used widely all over the world, notably in higher education and industry.

Q: Could you tell me about your role with ADL SCORM?

A: I’m involved in planning the ADL SCORM update: looking at the form the update will take, who we should talk to, the timeline, the tasks we need to do,  the technical direction, and then communicating those out to various people: vendors, users, and their organizations both here in the US and internationally. I’m also helping ADL upgrading its registry federation model for learning content.

Q: Can you share some details about the planned update to SCORM?

A:  SCORM has not been through a major update since 2004, although we made some minor, incremental changes along the way.  By talking with vendors and user communities, we are finding divergent views about what’s important and what’s lacking — those kinds of things. We know that technology has evolved on the Web.  We also know we have legacies out there: we have people procuring SCORM-based delivery systems today with the expectation that they’re going to be around in a number of years and that they are still going to work. So the objective is to produce an update that is grounded in the reality of what people want to use and what they actually have in place. We have to balance legacy support requirements with new capabilities and requirements. Our goal is to produce an updated, revised and harmonized version of SCORM by late 2011. We are talking to users, talking to adopters and to vendors to find out what things they are planning to put into their products in the short term – say the next 18 months or so. We may not be able to do everything we want to do in that  timeframe, but  our assumption is that things that don’t hit this cycle will get into the next cycle.

Q: What issues are you finding as you talk with people about the future of SCORM?

A: A number of things keep coming up: Web services, different content exchange models, content mark-up languages, assessments – those are the kinds of things that are on the list. What we are trying to do is develop a flexible model that indicates what we feel is important and that vendors can get behind. We also want organizations to be able to add features of their own in harmony with what we are doing. We know we won’t be able to do everything ourselves; we are relying on the community to bring their resources to bear on this and help us collectively move things forward.  We call it harmonization because we want to bring people together and move forward together.

Q: What’s prompted you to attend the Questionmark Users Conference?

A: We’re reaching out to a number of different vendors to find out what their users are interested in. We’ve been in discussions with Questionmark for awhile and this came up as an opportunity to talk with this user segment. Assessments standards are not currently in SCORM, but we know assessments and tracking are things people have said are very important. You have one of the best user communities there to talk to, so we want to talk with them and find out what they see as being important in SCORM as we go forward.

Q: Who should attend this session?

A: Anyone from the SCORM community.  We’ve got different communities that have different levels of adoption. Also, anyone who uses AICC, which has some elements in common with SCORM. They have a significant overlap in what they do and how they do it. We have been in direct communication with AICC about the harmonization and would welcome ideas from their users. Also, from the standpoint of specifications and standards, we’re interested in hearing from anyone with opinions about IMS QTI.

Q: What kind of input are you looking for during your session about the future of SCORM?

A: We want to know what people are currently doing; what problems they’re seeing; what they think in general about assessment and tracking. We’d like to learn more about where they do and don’t use SCORM and why. We’ll also ask what we can change in SCORM to make their lives better.

Come to the conference and let your voice be heard! You can register online or email conference@questionmark.com for further information.

This is the fourth of a series of  posts on standards that impact assessment.

The ADL SCORM standard rose out of an initiative by the US Department of Defence (DoD), who were large users of e-learning. They wanted to ensure that e-learning content could be interoperable and reusable, for instance to ensure that if e-learning content was developed by one vendor, it could run in another vendor’s environment.

The DoD has a track record of setting technical standards:  for instance  in the 1980s they helped popularize and make TCP/IP an effective standard. The DoD is also a very large customer for most companies in the learning and technology software industry. So when the DoD announced that it would only purchase e-learning software that worked with SCORM, the industry jumped quickly to support it!

One of the ways in which SCORM was made successful was by a series of Plugfests, where vendors could get together in practical labs and check that interoperability was possible in practice, not just in theory. These were well run events, a kind of technological speed dating, where each vendor could try out their compatibility with other vendors. It was great to have technical experts from each vendor in the room and be able to have many different LMSs all able to call our assessments.

In Questionmark Perception, to make an assessment run via SCORM, you use the Publish to LMS capability to create a content package, which is a small XML document that references the assessment. And as you can see in the screenshot below, you can choose from AICC and two flavours of SCORM. Once you’ve made the package, you simply upload it to a management system and participants can then be directed to it.

SCORM is used widely, both within the military and outside it. If you have a choice between AICC and SCORM, it’s often better to choose AICC (see my earlier post in this series), partly because SCORM has a potential security issue (see our past blog article). However, providing you are aware of this issue, SCORM can be a very effective means of calling assessments.

The ADL are currently reviewing SCORM and working out how to improve it, including potentially making it more useful for assessments. As part of their listening for this review, ADL’s technical advisor, Dan Rehak, who was one of the architects of SCORM, is running a session at Questionmark’s user conference in Miami in March to gain feedback on how SCORM could be improved. If you’re interested in influencing this standard to be better in future, this would be a great session to go to. Stay tuned here on the blog for a Questionmark Conference Close-up interview with Dan.

This is the second of a series of blog posts on assessment standards. Today I’d like to focus on the IMS QTI (Question and Test Interoperability) Specification.

It’s worth mentioning the difference between Specifications and Standards: Specifications are documents that industry bodies have agreed on (like IMS QTI XML), while Standards have been published and committed to by a formal legal body (like AICC or HTML). A Specification is less formal than a Standard but still can be very useful for interoperability.

Questionmark was one of the originators of QTI. When we migrated our assessment platform from Windows to the Web in the 1990s, our customers had to migrate their questions from one platform to the other. As you will know, it takes a lot of time to write high quality questions, and so it’s important to be able to carry them forward independently of technology. We knew that we’d be improving our software over the years and we wanted to ensure the easy transfer of questions from one version to the next. So we came up with QML (Question Markup Language), an open and platform-independent method of maintaining questions that makes it easy for customers to move forward in the future.

Although QML did solve the problem of moving questions between Questionmark versions, we met many customers who had difficulty bringing content created in another vendor’s proprietary format  into Questionmark. We  wanted to help them, and we also wanted to embrace openness and allow Questionmark customers to export out their questions in a standard format if they ever wanted to leave us. So we worked with other vendors within the umbrella of the IMS Global Learning Consortium to come up with QTI XML, a language that describes questions in a technology-neutral way.  I was involved in the work defining IMS QTI as were several of my colleagues: Paul Roberts did a lot of technical design, Eric Shepherd led the IMS working group that made QTI version 1, and Steve Lay (before joining Questionmark) led the version 2 project.

Here is a fragment of QTI XML and you can see that it is a just-about-human-readable way of describing a question.

Effective assessment often needs to be integrated with other systems. Some integrations are proprietary, but wherever possible Questionmark tries to integrate using technology standards, as these are longer lasting than proprietary solutions and allow us to build one solution which can work for many customers.

Over the years Questionmark has been involved in many standards initiatives, and I thought I’d share in a series of blog articles a personal perspective of some of the key standards that impact assessment.
I’ll start with the standard commonly called AICC or AICC HACP (more formally AICC AGR-10), which is used by learning management systems to call assessment content. Millions of Questionmark assessments are called each year via AICC, and it’s the most successful of all the standards we use.

The AICC is an aviation industry organization founded in 1988, the same year Questionmark. Airlines and airplane makers wanted a way to deliver computerized learning to help people maintain planes that could last the 20 years or so years that the planes themselves would last. The original AICC standard was file based but was soon updated to work over HTTP.

In an assessment context, the AICC standard allows launching and tracking of an assessment:

1.  A Calling Application (for instance an LMS) calls an assessment system saying that it wants to start an assessment.

2.  The assessment system asks the Calling Application for the details of which participant and which assessment.

3.  The Calling Application replies and the assessment starts.

4.  At the end of the assessment, the score is passed back to the Calling Application for tracking.

A key reason the AICC standard is robust and successful is that there is direct server-to-server communication. The two pieces of software communicate directly by HTTPS, and so there is no possibility of disruption or interference by anything at the participant workstation.

I was introduced to the AICC standard in the 90s by Bryan Chapman (now an e-learning analyst) and Questionmark supported it as a way of making it easy for people with LMSs to call Questionmark Perception assessments. We first became certified to the AICC standard in 1999 and we’ve been re-certified several times since. The great thing about the AICC standard is that it really works: Because it’s been tried and trusted over many years, I can think of at least 25 different vendors that Questionmark has interoperated with using this standard, ranging from PeopleSoft and Sun down to much smaller vendors.

I prepared a new segment on Understanding eLearning Standards. This segment addresses the “how” of elearning standards, and specifically run-time communication using the common AICC HACP specification. [Don't worry SCORM fans, there will be another segment focusing on the SCORM runtime.]

Standards fans (and hockey fans) are likely to appreciate the analogies used to explain a run-time environment in general. The video also steps through the lifecycle of an activity running in an LMS environment. Then I drill down to the specific of AICC, including both the common browser-to-LMS and the compelling server-to-server uses of AICC HACP.

Finally, the segment closes with a review of key resources from the AICC web site to help you make the most of AICC HACP.

By the way, here is an extra resource for members of the Questionmark Software Support Plan Community. There is a great Knowledge Base article on customizing the Perception v4 PIP file for AICC. This article shows how you can use a custom PIP file to utilize additional demographic or custom variables from an AICC compatible LMS. Check it out.

Stay tuned to the Questionmark Blog for the next segment that will address SCORM Run-Time Communication.

I’ve prepared a video podcast which is your introduction to key interoperability standards for elearning. It also serves as my introduction to video podcasts. Your feedback on both the content and the style will be put to use as I continue the series—so please post comments or send email.

The video for Part 1 provides a quick overview of the need for interoperability standards, the names of the keys standards, and the types of interoperability they support. Part 1 addresses AICC, ADL SCORM, IEEE LTSC and IMS specifications at a high level. It introduces the concepts of run-time communication, content packaging, and meta-data.

I hope you find it a good refresher if you are already somewhat knowledgeable about these standards, and an excellent introduction if you are new to most of this.

Posted by John Kleeman

A key objective for Questionmark Perception has been to make it an open system that handles integrations easily. Assessment isn’t usually standalone; most organizations need to integrate it with other organizational systems. There are many ways to integrate with Perception, including via our support of standards such as AICC, HR-XML and SCORM, but where standards are not available we recommend integration via our QMWISe web services.

Although web services are routine today, Questionmark adopted them very early: June 6th, 2009, marks the 7th anniversary of Questionmark’s web services, which we call QMWISe. (See our 2002 press release here.)

Two great advantages of web services are that you can call them from almost any platform or system and they are independent of the technology used. So you can code web services in almost any programming language or environment and interface with Questionmark Perception.

Another beauty of web services is that code written back in 2002 will still work in 2009,and code written today should still work in 2016! In the last seven years, there have been very substantial changes to the Questionmark Perception database format and to the user interfaces, but the APIs (Application Programming Interfaces) remain the same. And exactly the same code written then to call QMWISe will still work now. We have ambitious plans to continue developing Questionmark software in new ways, but code our customers write today for QMWISe will still work in the future.

Back in 2002, there were 37 web services methods. Over the years, we’ve added lots more methods and there are now 109. Example web services methods are to create a participant, schedule a participant or give a URL to get access to an assessment.

Many of our customers use QMWISe to integrate with Perception, so that as Perception versions change, their code can remain safe. We or our partners have also used QMWISe to build connectors to many other systems, including Blackboard, Moodle and uPortal. We also call QMWISe within our own software. For instance, Questionmark to Go passes all its results back via web services, and in the future we’ll be trying to use QMWISe more within other code–to “eat our own dog food” and ensure that QMWISe is fully able to be mission critical. By using web services within our own code, we will be driving QMWISe forward to cover more capabilities and so open up the platform to support a wide range of solutions integrated with third party applications.

One key lesson that we’ve learned over time with web services is that commitment and continuity are vital. No one wants to interface with a system that will change. And you need to have good documentation with examples, good scalability and good diagnostics–for instance a log of all SOAP traffic. We recommend that other developers consider making web services available from their own systems: it’s an excellent way of integrating.

In the future we’ll be announcing further improvements to QMWISe that should make it more useful for developers and provide easier ways for customers to integrate with Perception. Questionmark strongly recommends that anyone developing integration into our software uses our web services. We welcome questions, comments and suggestions for improvements, so let us know what you think!

Posted by Tom King

My earlier post, The Importance of Security and Integrity of Performance Data addressed a specific emerging SCORM security issue. It also raised the issue of “Defense in Depth” as an approach for improving security. Here are some defense in depth approaches you can use right now to increase security and decrease vulnerability.

Key ways to reduce vulnerability and improve security.

• Audit trails and accountability. Have a second source of data to cross-check. Ideally this data should be automatically collected. Data sent to a SCORM or AICC LMS is also sent to a Questionmark Perception server via a different data conduit.
• Secured Communication. Transfer responsibility for the result data to a server. Questionmark’s secure server-to-server implementation of AICC does this.
• Increased Client/Browser Security. Reduce the attack surface of the runtime. Use a Secured Browser that disables or limits functionality not directly needed for the primary activity. Questionmark Secure is a browser that does this for AICC or SCORM.
• Direct Proprietary Communication. This approach works by centralizing the chain-of-custody for the data to one trusted provider. Questionmark Perception can manage the process completely from authoring to scheduling to delivery to reporting.

Audit trails. Keeping parallel records such as with a double-entry accounting system is one way to achieve an audit trail. Having such an audit trail is key to identifying and recovering from errors or misdeeds. Questionmark provides capabilities for such an audit trail through both its SCORM and its AICC implementations. Perception achieves increased security and this audit trail by sending data to the LMS using the SCORM or AICC standard and, in parallel, sending data directly to the secure Perception server database. In the case of an error or misdeed, the LMS system results and the results in the secured Perception database can be compared to recover from either a security breach or an error.

Secured Server-to-Server Communication. In the cheatlet exploit, the openness of the published SCORM API and the browser JavaScript layer are used to inject false data from the client side. One way to increase the security is to remove this client side vulnerability and use AICC instead of SCORM. The innovative Perception server-to-server implementation of the AICC HACP specification demonstrates this, by having the browser relay minimal data to the Perception server. The client is not capable of directly injecting falsified overall score data. The Perception server is ultimately responsible for judging response and data communication with the LMS, not the browser client.

In 2002, Paul Roberts of Questionmark identified and described the risks of the client-side API (see Security Issues with the JavaScript API, Paul Roberts, 2002 on the AICC web site). He urged the AICC to continue to support the HACP protocol because of the value of the increased security enabled with a server-to-server AICC implementation. The diagram below helps explain this communication.

Increased Browser Security. As currently implemented, this exploit relies on user access to the UI to open a bookmark. Changes to the launch environment (browser) can reduce this vulnerability. The Questionmark Perception Secure Browser is a commercialized browser solution built for the rigorous requirements of high-stakes testing environments. When a participant takes an online assessment using Questionmark Secure, the secure browser displays the HTML content of the assessment, but disables key functions such as task-switching, right click options, screen captures, menus and printing. There simply isn’t a means to access a menu or bookmark to trigger.

Direct Proprietary Communication In this scenario, one trusted party is responsible for the full span of access, delivery, and results. It does run somewhat contrary to cybersecurity practice of published protocols and specifications that can bear wide scrutiny. It can also undermine interoperability, something near and dear to my heart. In the long run, I believe you’ll find Questionmark moving in directions that addresses these type of concerns.

However, there are many valid circumstances where the values of single party chain of custody and trusted relationship trumps other concerns. High stakes test are often the prime case for this, and it is critical to expand cyber-defense-in-depth with adjunct security measures (such as tight control of source materials, exam monitors, proctors/invigilators).

Work-around versus defend-against. Finally, as an exercise for the reader, you may consider reading the the two ADL workarounds published April 2, 2009. You’ll find that the excerpt on Securing Your Assessments provides a means of masking the location of answer-judging source code sent to the client by some systems. While useful, it doesn’t provide the same security and depth of defense as other approaches. Consider for instance using Questionmark Secure (prevents ‘view source’) with the Perception SCORM implementation (adds audit trail) and Perception server-side evaluation logic (secures the evaluation logic on the server-side). That is defense in depth. One might even replace SCORM with AICC in this case for additional security in addition to or in lieu of Questionmark Secure.

Whenever faced with security concerns regarding the possibility of cheating, abuse or data integrity, I encouraged you to think about defense in depth and the role of all the components in security.