This is the second of a series of blog posts on assessment standards. Today I’d like to focus on the IMS QTI (Question and Test Interoperability) Specification.

It’s worth mentioning the difference between Specifications and Standards: Specifications are documents that industry bodies have agreed on (like IMS QTI XML), while Standards have been published and committed to by a formal legal body (like AICC or HTML). A Specification is less formal than a Standard but still can be very useful for interoperability.

Questionmark was one of the originators of QTI. When we migrated our assessment platform from Windows to the Web in the 1990s, our customers had to migrate their questions from one platform to the other. As you will know, it takes a lot of time to write high quality questions, and so it’s important to be able to carry them forward independently of technology. We knew that we’d be improving our software over the years and we wanted to ensure the easy transfer of questions from one version to the next. So we came up with QML (Question Markup Language), an open and platform-independent method of maintaining questions that makes it easy for customers to move forward in the future.

Although QML did solve the problem of moving questions between Questionmark versions, we met many customers who had difficulty bringing content created in another vendor’s proprietary format  into Questionmark. We  wanted to help them, and we also wanted to embrace openness and allow Questionmark customers to export out their questions in a standard format if they ever wanted to leave us. So we worked with other vendors within the umbrella of the IMS Global Learning Consortium to come up with QTI XML, a language that describes questions in a technology-neutral way.  I was involved in the work defining IMS QTI as were several of my colleagues: Paul Roberts did a lot of technical design, Eric Shepherd led the IMS working group that made QTI version 1, and Steve Lay (before joining Questionmark) led the version 2 project.

Here is a fragment of QTI XML and you can see that it is a just-about-human-readable way of describing a question.

Effective assessment often needs to be integrated with other systems. Some integrations are proprietary, but wherever possible Questionmark tries to integrate using technology standards, as these are longer lasting than proprietary solutions and allow us to build one solution which can work for many customers.

Over the years Questionmark has been involved in many standards initiatives, and I thought I’d share in a series of blog articles a personal perspective of some of the key standards that impact assessment.
I’ll start with the standard commonly called AICC or AICC HACP (more formally AICC AGR-10), which is used by learning management systems to call assessment content. Millions of Questionmark assessments are called each year via AICC, and it’s the most successful of all the standards we use.

The AICC is an aviation industry organization founded in 1988, the same year Questionmark. Airlines and airplane makers wanted a way to deliver computerized learning to help people maintain planes that could last the 20 years or so years that the planes themselves would last. The original AICC standard was file based but was soon updated to work over HTTP.

In an assessment context, the AICC standard allows launching and tracking of an assessment:

1.  A Calling Application (for instance an LMS) calls an assessment system saying that it wants to start an assessment.

2.  The assessment system asks the Calling Application for the details of which participant and which assessment.

3.  The Calling Application replies and the assessment starts.

4.  At the end of the assessment, the score is passed back to the Calling Application for tracking.

A key reason the AICC standard is robust and successful is that there is direct server-to-server communication. The two pieces of software communicate directly by HTTPS, and so there is no possibility of disruption or interference by anything at the participant workstation.

I was introduced to the AICC standard in the 90s by Bryan Chapman (now an e-learning analyst) and Questionmark supported it as a way of making it easy for people with LMSs to call Questionmark Perception assessments. We first became certified to the AICC standard in 1999 and we’ve been re-certified several times since. The great thing about the AICC standard is that it really works: Because it’s been tried and trusted over many years, I can think of at least 25 different vendors that Questionmark has interoperated with using this standard, ranging from PeopleSoft and Sun down to much smaller vendors.

Posted by Steve Lay

Last month I wrote about our new Community Edition program under which we’ll be releasing selected integration products as free-to-download Community Editions distributed under open source licenses.

Although this program represents an exciting new way to involve the development community, Questionmark is no stranger to working with open source. Moodle is a popular learning management system that is developed by an active community of open source developers. Questionmark has always distributed its connector products for Moodle under a similar open source license, so it is no surprise that our new connector for Moodle 1.9 is available as a Community Edition.

The connector is written in the widely-used PHP system used by Moodle itself and uses QMWISe (Questionmark Web Integrated Services Environment) and PIP (Perception Integration Protocol) to allow Perception tests to be included as activities directly within Moodle courses.

You can find our more information about the connector and the development project behind the Community Edition from our new developer site: developer.questionmark.com. On the site, you’ll find information about our Application Programming Interfaces (APIs) and all of our Community Edition projects. The Community Edition Moodle Connector page on the development site also links to the development project, which is hosted on the popular SourceForge system where you can download the latest version and even browse the code!

And finally, we’ve used Moodle as a development platform for the developer site itself, which means that we can demonstrate the connector working directly from the site. Look for the “Try It Out!” section on the Community Edition Moodle Connector page and see if you can answer my quick quiz on the main features of our Community Edition release program.

Last week I wrote about tips for protecting  intellectual property.  It’s equally important keep people from cheating on tests, so here are three pointers on that subject.  I’ll be following this up with more tips in future posts.  Leave me your comments; we can always add to lists like these!

.Screening tests

Consider givinig a small pre-screening test to prevent people from taking an assessment that is beyond their current ability level. If a participant can‘t answer a certain number of these questions correctly they will not be allowed to see the remainder of the assessment. When the time does come for them to take the test,  they will not have already seen its content.

Candidate agreements

Candidate agreements or examination honor codes require  a participant’s agreement before they start an assessment, say by clicking on an “OK “ or  “Yes” button after reading the exam’s code of conduct.

The code might say something like this:  “I agree to answer the questions on this assessment without obtaining assistance from another person or via electronic means. I agree to not to share my answers with anyone during or after the exam. I further agree to not memorize or otherwise steal the intellectual property contained in this exam. I accept that if any of these conditions are violated, my exam results will be set to a zero, I will not be able to retake the exam for a period of 10 years, and I may be charged with a crime under regional laws.”

Here are some topics you might want to cover in a candidate agreement:

• The test vendor will have the option to terminate the assessment if suspicious behavior is detected
• The candidate must abide by the rules of the test center, organization, or program
• The candidate will not provide false ID or false papers
• The candidate cannot take the test on behalf of someone else
• The candidate will not engage in cheating in any form
• The candidate will not help others cheat by disclosing information about the assessment
• The candidate will not use aids that are not allowed
• The candidate will not solicit someone else to take the test
• The candidate will not cause a disturbance in the testing center
• The candidate will not tamper with the test center in any way
• The candidate will not share information

Limiting content exposure/leakage

In order to limit the amount of question content being shown to a participant at any given time, think about using question-by-question templates. These present questions one at a time to participants so that exam content is not completely exposed on screen. Participants who may intend to take pictures of the exam content or otherwise steal intellectual property will not be able to do so all at once.

There are many fine resources for learning how to prevent cheating. Two of thes are books by Dr. Gregory Cizek:  Cheating on Tests: How to Do It, Detect It and Prevent It and Detecting and Preventing Classroom Cheating: Promoting Integrity in Assessment.

There’s also our white paper: “Delivering  Assessments Safely and Securely,” and of course this blog! Watch for more security tips in my future posts.

Protecting the intellectual property contained in a test or exam is essential, not only because of the time, effort and cost of creating assessments but also because IP theft undermines the accurate measurement of knowledge and skills.

Protecting intellectual property protects the credibility of tests. Here are four tips for helping to ensure the security of intellectual property:

Create and administer multiple test forms

Rather than having only one form of the assessment being administered, delivering multiple forms of the same exam can help limit item exposure. This method also allows for the possibility of interspersing large-scale integrated beta test questions within the forms to collect psychometric information on newly developed questions.

Restrict and control administration of beta test items

Beta testing questions is an important part of high-stakes assessment, ensuring the psychometric quality of questions before they appear on actual assessments. However, it is vital that a well conceptualized beta test model is in effect to limit the exposure of newly developed questions to participants.

Update exam forms periodically

Letting exam forms become stale can over-expose questions to participants, increasing the likelihood of IP theft. An organization could consider retiring old exam forms and turning them into exam prep materials that can be sold to participants. In this way, participants could periodically expect new practice questions.

Produce exam prep materials

Organizations should consider making exam prep materials available to participants before an assessment. This will help reduce the demand for participants to try to obtain exam questions via illegal means as they will have access to the type of questions that will be asked on the actual assessment.

For more details on this subject, plust information about various means for deploying a wide range of assessment types with assurance, download our White Paper: Delivering Assessments Safely and Securely.

Assessments provide a valuable tool for helping organizations properly design effective and useful learning experiences. Doing so involves a five-step process.

Step 1: Define the objectives. An objective might be to increase customer satisfaction, reduce error rates or improve safety.

Step 2: Ask what knowledge and skills are required to meet the objectives. In a college or university course on organic chemistry, for example, it’s important to ask what are the knowledge and skills required to comprehend those concepts and use them. The answers to that question will help the professor establish the topic structure to define the knowledge, skills, abilities and attitudes required to meet the objectives.

Step 3: Run a needs analysis survey or skills gap survey. Here, people take a needs assessment to reveal the knowledge and skills they already have as well as what they still need. A gap analysis can be derived from the difference between what is required and people’s current knowledge and skills.

Step 4: Develop a learning plan. That plan will describe the learning objectives and explain how the plan will be administered. The learning objectives will guide the production of learning materials and assessments. Facilitating the learning process might involve instructor-led training, coaching by managers, or e-learning courses.

Step 5: Conduct a pre-learning assessment. The pre-learning assessment will have two purposes: to create intrigue about the course and  to guide each participant to the right learning experience. Advanced and novice learners will require different approaches.

To learn more about  how instructors and organizations can use assessments to improve learning, download the white paper Assessments through the Learning Process.

Posted by Greg Pope

I had a great time putting together an eight-part series on Item Analysis Analytics for this blog and was pleased with the interest it received.

When a reader asked if it would be possible to present all the posts in a single document I thought hey, let’s present the content of these articles in the form of a Questionmark White Paper! So here it is for you to download with our compliments.

I hope the paper helps you in your efforts to create test questions that make the grade!

In talking with some of our customers last week, I was reminded how valuable it can be to offer participants topic based feedback.

Obviously, everyone wants to know whether they’ve passed or failed a test. And most people look at their feedback on questions they got wrong, to understand how to improve. But you can get a single question wrong for many different reasons including misunderstanding the question, making a mistake or having a tiny gap in knowledge. However, if you score weakly in a topic area, it very likely means you have a weakness in that area that needs addressing.

In many ways, topic feedback is the Cinderella in the feedback world. Everyone expects assessment level feedback and item level feedback (perhaps it’s unfair to call them ugly sisters because they are useful and valuable), but there is a huge and often untapped learning value in topic feedback. For pre tests, post course tests, quizzes during learning and practice tests particularly, topic feedback is vital.

Suppose someone is taking an assessment in health and safety and they score 66% and this is a passing score. Sounds good! But what happens if, as in the screenshot below, they’ve scored very well in some topics and poorly in others?

In this example, the fact that someone is very weak in electrical safety could well be concerning. (Don’t let them set up the lighting for the Holidays Ball!)

It’s obvious that you want to give people feedback on the topic level, but in many tools this isn’t as easy to set up as it should be. Questionmark Perception can be a fairy godmother for topic based feedback. There are lots of easy-to-use capabilities to present topic based feedback. Here are some links to support resources to help you create topic feedback in Perception.

• You can easily create topic outcomes with feedback for different topic scores in Authoring Manager
• You set these as standard at the topic level, but can override to adjust on a per assessment level
• You can also make a topic a prerequisite for passing an assessment (for instance to prevent someone passing an assessment unless they get 60% or a specified score in key topics).
• If you want to display only some topics in the list and not all, for instance if some of your topics aren’t meaningful to the participant, you can define which topics are reported on.
• And then the feedback is displayed to participants at the end of assessment easily as in the screenshot above.

I hope you find this useful in getting your topic feedback working and helping your learners achieve their full potential.

Is it possible to build the perfect assessment design? Not likely, given the intricacies of the learning process! But a white paper available on the Questionmark Web site helps test authors respond effectively to the inevitable tradeoffs in order to create better assessments.

Measuring Learning Results, by Dr. Will Thalheimer of Work-Learning Research, considers findings from fundamental learning research and how they relate to assessment. The paper explores how to create assessments that measure how well learning interventions are preparing learners to retrieve information in future situations—which as Will states it is the ultimate goal of training and education.

The eight bits of wisdom that conclude the paper give plenty of food for thought for test designers. You can download the paper to find out how Will arrived at them.

1. Figure out what learning outcomes you really care about. Measure them. Prioritize the importance of the learning outcomes you are targeting. Use more of your assessment time on high-priority information.

2. Figure out what retrieval situations you are preparing your learners for. Create assessment items that mirror or simulate those retrieval situations.

3. Consider using delayed assessments a week or month (or more) after the original learning ends—in addition to end-of-learning assessments.

4. Consider using delayed assessments instead of end-of-learning assessments, but be aware that there are significant tradeoffs in using this approach.

5. Utilize authentic questions, decisions, or demonstrations of skill that require learners to retrieve information from memory in a way that is similar to how they’ll have to retrieve it in the retrieval situations for which you are preparing them. Simulation-like questions that provide realistic decisions set in real-world contexts are ideal.

6. Cover a significant portion of the most important learning points you want your learners to understand or be able to utilize. This will require you to create a list of the objectives that will be targeted by the instruction.

7. Avoid factors that will bias your assessments. Or, if you can’t avoid them, make sure you understand them, mitigate them as much as possible, and report their influence. Beware of the biasing effects of end-of-learning assessments, pretests, assessments given in the learning context, and assessment items that are focused on low-level information.

8. Follow all the general rules about how to create assessment items. For example, write clearly, use only plausible alternatives (for multiple-choice questions), pilot-test your assessment items to improve them, and utilize psychometric techniques where applicable.