Revision History in Questionmark Live

Revision history is an collaborative authoring capability of Questionmark Live. Here are the basics:

What it does: This feature enables users to track and manage edits made to questions:

    • View a question’s full revision history
    • Compare different versions side-by-side with marked-up changes
    • Roll back to previous versions of questions to undo edits made by others

Who should use it: Subject matter experts can use this feature to easily review, keep track and roll back to a previous version of the question.

How it looks: Each revision of a question is listed in the table. Where:

  • Revision – identifies how many revisions have been created
  • Version ID – identifies the version of the question (Used to identify which version of the question any rollback has been reverted to)
  • Modified On – states the date that the revision was made
  • Modified By – highlights who made the actual revision
  • Change Type – describes what the revision entailed
  • Comments – lists the notes made by the author that revised the question

By selecting a revision you can view the question. It is possible to select more than one revision to compare the differences. To roll back to a previous revision, select the revision and click Rollback.

Working with ADL on enabling developers to create learning activity streams

Posted by Jim Farrell

Questionmark has decades of commitment to designing our product in accordance with industry standards including AICC, ADL SCORM, .NET, HR-XML, and IMS QTI.

Next week at the eLearning Guild’s mLearnCon conference, Questionmark will continue that commitment by supporting a new ADL (Advanced Distributed Learning) initiative that we think  will help drive how we learn for years to come.

Why are we so excited about the ADL initiative? At Questionmark, we love how the 70+20+10 model explains that 90 percent of learning is done outside of the formal learning environment. We learn by doing and by working with others. In this world, you are challenged to know if people are learning and if their performance is better because of the learning.

In other words, how do you track learning without launch in a formal learning situation?

Welcome to the ADL’s initiative, Project Tin Can. Although currently working under just a project title, this initiative gives content and app developers the power to create activity streams that can be sent to a Learning Record Store (LRS) and later analyzed. The activity streams are made up of an actor, verb, and object — as in this statement: “I did this.”

Here are some examples of what the activity streams could look like:

  • John Smith engaged the knowledge base article, “How do I embed a video”
  • Tom Brady attempted the simulation on How to Throw a Football
  • Sandy Pine visited the customer Apple Computer

None of these activities are typical launch-and-track activities, but they could help you understand what makes a top-performing contributor.

Questionmark is extremely excited to be part of ADL’s initiative and will be demonstrating the production of activity streams from a Questionmark assessment at mLearnCon.

Please stop by and say hi if you are going to be in San Jose next week.


Keeping up with Software Security

Posted by Steve Lay

Readers of this blog will be familiar with many of the techniques that can be used to improve the security of your testing programmes. Part of that picture involves securing the software you use. With Cyber-warfare very much in the news I thought I’d take this opportunity to tell you a little about some of the challenges involved in keeping up with the bad guys.

It seems obvious that we all want the software we use to be secure, but security isn’t just a feature that can be implemented once like a new menu option in an application. The security landscape is constantly changing and software needs to change to remain secure.

Anyone who uses a PC or a web browser will know that software updates are pushed out routinely. Even my mobile phone receives updates on a regular basis. It isn’t always easy to tell why software is being updated, but in many cases the updates do contain important security fixes that address newly discovered vulnerabilities.

To give just one example of why this environment is so challenging, in late December 2011, the US Computer Emergency Readiness Team (US-CERT) issued a bulletin concerning a very basic weakness in many web applications (See The weakness involved the parameters passed to web applications by your browser when you fill in an online form. A malicious user could cause the web server to slow down by sending it carefully selected parameters. There was no threat to the confidentiality of the information on the server, but there was a threat to the service availability. Suddenly it was possible for a suitably motivated individual to take almost any website offline. This type of attack is known as a “Denial of Service” or DoS attack.

Fortunately, software developers quickly addressed the problem and patches were distributed for affected systems including the frameworks we use. The team that runs Questionmark OnDemand was quick to act and the patches were tested and deployed very rapidly.

Although the breadth of systems affected was unusual in this case, in many respects this was a typical incident. Nobody sets out to make software that is insecure, but as we learn more about computing we discover new ways that systems can be misused. Keeping software up to date is critical to maintaining security. Users of Questionmark On Demand can be confident that our team of system administrators puts a very high priority on keeping the service current. You can read more about the trustable platform we use for Questionmark OnDemand in our white paper, Security of Questionmark’s OnDemand Service.

I’ve chosen to highlight a fairly dramatic case in this blog post, but sometimes we do get more warning. Cryptography is an important tool in the security arsenal. Computer scientists and mathematicians develop the algorithms on which cryptography depends. Cryptography doesn’t make it impossible to read confidential information; it is designed to make it impractical with the technology available today. As computers get faster, the codes become more vulnerable until eventually they have to be retired and replaced with new, stronger codes. You can actually see this effect in the results of the RSA Challenges. The RSA offered prizes to people who could crack codes of increasing strength. Just looking at the years in which each prize was awarded gives a feeling for the dynamic nature of this field: 1991, 1992, 1993, 1994, 1996, 1999, 2003 and the last prize was awarded in 2005 (US$20,000).

This may all sound esoteric, but behind the scenes our developers are working on issues like these to ensure that Questionmark technologies continue to meet with the strictest requirements, and that we stay ahead of the code-breakers.

Hierarchical topics coming to Questionmark Live

Posted By Doug Peterson

Questionmark Live browser-based authoring has some really exciting new features coming out soon. The latest: Hierarchical topics and assessment creation enhancements.

Hierarchical topics

  • The Topics tab will feature a single-page interface where the Topic tree and questions list are shown on the same page, saving time when switching between Topics — just click on the Topic you want to view and the questions list immediately updates, displaying the questions contained within that particular Topic without having to leave the screen.
  • Topic hierarchies will be maintained when exporting and importing Qpacks —  compressed files containing all the questions, wording and resources used in a Question Set.
  • You will be able to share Topics at any level in the Topics hierarchy  with other Questionmark Live users (i.e., you could share a particular Topic without sharing a Topic at a higher or lower level in the hierarchy).

Assessment creation enhancements

We’re also making  a number of changes to assessment creation in Questionmark Live, so that you can create a Test, Exam, or Quiz, in addition to a Course Evaluation survey.

Existing Questionmark Live assessments will be maintained as Tests (existing Course Evaluation surveys will remain Course Evaluation surveys). The new assessment creation process will allow you to create and re-order multiple question blocks and assessment outcome blocks; configure settings such as time limit, security level and feedback level for an assessment,  and choose delivery options like  Save-As-You-Go.

Stay tuned on the blog for more details about these and other great features.

Meanwhile, if you’d like to become more familiar with Questionmark Live,  check out this webinar on June 27, 2012 – 12:00 PM (EDT).

You can try out Questionmark Live for yourself by clicking here.


Webinar: Using principles of enterprise architecture to build assessments

Posted by Joan Phaup

Tom Metzler

Looking for ways to develop assessments more efficiently?

You can get some inspiration on that front from our next Questionmark Customers Online webinar:

Using Principles of Enterprise Architecture to Build Assessments.

Tom Metzler, an enterprise architect and senior education consultant at TIBCO Software, Inc., will tell how the compnay’s certification team has used well-established software architecture methods to implement a more efficient assessment development process. His presentation will show how the team capitalized on documenting their knowledge assessment program to improve  assessment development. And he’ll describe how they achieved their anticipated return on investment.

You can learn more about Tom’s presentation by reading my previous Q&A interview with him prior to this year’s Questionmark Users Conference. For webinar details and free online registration, visit our web seminars page and scrolling down to Customers Online.




Ten benefits of quizzes and tests in educational practice

Posted by John Kleeman

I’d like to share ten benefits proposed by psychology experts Henry L. Roediger III, Adam L. Putnam and Megan A. Smith in a recent paper, “Ten Benefits of Testing and Their Applications to Educational Practice”.

Here is my summary and understanding:

1. Retrieval aids later retention. There is clear evidence from psychological experiments that practicing retrieval of something after learning it, for instance by taking a quiz or test, makes you more likely to retain it for the long term.

2. Testing identifies gaps in knowledge.

3. Testing causes students to learn more from the next study episode. Essentially it reduces forgetting which makes the next related study area more productive.

4. Testing produces better organization of knowledge by helping the brain organize material in clusters to allow better retrieval.

5. Testing improves transfer of knowledge to new contexts. There are several experiments referenced in the paper where tests and quizzes help transfer and application of knowledge.

6. Testing can facilitate retrieval of material that was not tested. Surprisingly there are circumstances where quizzes or tests, particularly if delayed, can help people retrieve/retain information that was related to that asked but not actually asked in the questions.

7. Testing improves metacognitive monitoring – by giving students scores or self-assessments, they can better predict their knowledge and be more confident about what they know and what they need to know.

8. Testing prevents interference from prior material when learning new material. If you have a test after learning one set of material before learning another set of material, it can make it less likely that the second session will

9. Testing provides feedback to instructors and lets them know what is learned or what is not.

10. Frequent testing encourages students to study. Having frequent quizzes and tests motivates study and reduces procrastination.

You can see their paper “Ten Benefits of Testing and Their Applications to Educational Practice” in Psychology of Learning and Motivation, Vol 55. It’s also available for download on Professor Roediger’s publications page, in the list of papers from 2011, at