Our first Beyond Multiple Choice: Nine Ways to Leverage Technology for Better Assessments Web seminar met with an enthusiastic response: it filled up almost instantly! So we are offering the same seminar again on Wednesday, December 16th, at 3 p.m. Eastern Time.

Assessments play a vital role in measuring people’s knowledge, skills and attitudes. They also help organizations improve performance, manage workforce competencies, and ensure regulatory compliance. How can you create and assessments that produce appropriate, actionable results? What can you do to ensure the quality of questions and the security of assessments all the way from authoring and scheduling to administration, reporting, and analysis? How can you make the best use of online authoring, reporting, analytical, and security tools? These and many other questions will be addressed in this free, hour-long session, which will include opportunities for you to ask questions of your own.

We welcome you to learn more about this Webinar and register online.

This is the second of a series of blog posts on assessment standards. Today I’d like to focus on the IMS QTI (Question and Test Interoperability) Specification.

It’s worth mentioning the difference between Specifications and Standards: Specifications are documents that industry bodies have agreed on (like IMS QTI XML), while Standards have been published and committed to by a formal legal body (like AICC or HTML). A Specification is less formal than a Standard but still can be very useful for interoperability.

Questionmark was one of the originators of QTI. When we migrated our assessment platform from Windows to the Web in the 1990s, our customers had to migrate their questions from one platform to the other. As you will know, it takes a lot of time to write high quality questions, and so it’s important to be able to carry them forward independently of technology. We knew that we’d be improving our software over the years and we wanted to ensure the easy transfer of questions from one version to the next. So we came up with QML (Question Markup Language), an open and platform-independent method of maintaining questions that makes it easy for customers to move forward in the future.

Although QML did solve the problem of moving questions between Questionmark versions, we met many customers who had difficulty bringing content created in another vendor’s proprietary format  into Questionmark. We  wanted to help them, and we also wanted to embrace openness and allow Questionmark customers to export out their questions in a standard format if they ever wanted to leave us. So we worked with other vendors within the umbrella of the IMS Global Learning Consortium to come up with QTI XML, a language that describes questions in a technology-neutral way.  I was involved in the work defining IMS QTI as were several of my colleagues: Paul Roberts did a lot of technical design, Eric Shepherd led the IMS working group that made QTI version 1, and Steve Lay (before joining Questionmark) led the version 2 project.

Here is a fragment of QTI XML and you can see that it is a just-about-human-readable way of describing a question.

My previous post offered four tips on making your assessments more secure and preventing cheating.  Aside from verifying IP addresses and running a Trojan horse or stealth items to help detect whether a participant has memorized the answer key, there are some physical actions you can take to avoid the problem and reduce the temptation to cheat.

Proper seating arrangements for participants

Seating participants with adequate space between them and giving them limited ability to see another participant‘s screen or paper are important strategies for enhancing test security. The proctor should be aware of cheating techniques such as the ―”flying V” seating arrangement where the “giver” at the point of the V feeds information to a number of “receivers” behind them. The givers and receivers can communicate in a number of ways, using sign language, dropping notes on the floor, etc. (Dr. Gregory Cizek’s book “Cheating on Tests: How to Do it, Detect it, and Prevent it,” will tell you  more about this and other aspects of cheating.)

Example of the “flying V” answer copying formation (Cizek, 1999):

Using unique make-up exams

Many organizations offer make-up exams for participants who were sick or had legitimate excuses for not being able to take an assessment at the scheduled date and time. If you use the same exam that was administered at the scheduled date and time for their make-up exam, you open yourself to risks of the exam form being compromised. Sometimes the make-up exams are not administered in the same strict proctored environment as the scheduled exam, allowing participants the opportunity to cheat or steal content.

Using more constructed response questions

Constructed response questions, like essay or short answer questions, provide less opportunity for participants to cheat because they require them to produce unique answers to questions. There is no answer key to steal, and participants who copied other people’s constructed response answers are easily identified via a side-by-side comparison of answers.

I hope you enjoyed this three part series on preventing cheating.  You will find more information about  various means for deploying many different types of assessments in our white paper, “Delivering  Assessments Safely and Securely.”

We decided at our last U.S. Users Conference in Memphis that it would be good to set aside a special place for attendees to meet with our support technicians one-on-one. Thus was born Tech Central, which was such a hit that it’s become a regular part of our conferences.

As we get ready for our 2010 Users Conference in Miami next March, I asked Ed Bell, our technical services manager, a few questions about this recent addition:

What’s Tech Central? Tech Central allows customers at our Users Conferences to meet one-on-one with the technicians. We sit down with them for 20 minutes or half an hour and work through issues we would normally talk about on the phone. Tech Central gives us a dedicated space for meeting with people. Now they know exactly where to find us instead of hoping they’ll be able to find us in passing.

How did customers respond to the first Tech Central in Memphis? You couldn’t ask for a better response. The customers could see that we really cared. The first day was sparsely attended, but once word spread we got more and more visitors. Some of them would check in with their office after visiting us and then come back again with follow-up questions. About three-fourths of the people who attended the conference came to see us. Tech Central was such a hit that we took it to the European Users Conference in England this fall.

How was Tech Central for Techs? It was hectic because we had so many people coming through, but we thought it was great! Sometimes people were queued up because it was so popular. Luckily we had a really large space with WIFI, so people could email while they waited. What made it so wonderful for us was that it gave us a chance to really connect with our customers as people.  That face-to-face contact makes a big difference in our relationships with customers. When they call us on the phone, we know who they are!

What are you looking forward to most about the next Users Conference? Reconnecting with people I’ve met in the past! I enjoy seeing customers at Breakfast Briefings and our conferences. That personal bond you build from being with people is wonderful. I also look forward to seeing the customers’ reactions to the new features in Questionmark Perception,  and I always like hearing customers’ success stories and seeing how enthusiastic they are. People generally call tech support when they have a problem, and it’s our job to take care of that problem, so hearing about successes does us a lot of good! And those of us who attend the conference share those success stories with our colleagues, which gives them a boost, too.  I can’t wait for 2010!

If you can’t wait either, here’s a gentle reminder that earlybird-registration ends December 4th.

You can learn more about the 2010 Questionmark Users Conference at www.questionmark.com/go/conference.

Effective assessment often needs to be integrated with other systems. Some integrations are proprietary, but wherever possible Questionmark tries to integrate using technology standards, as these are longer lasting than proprietary solutions and allow us to build one solution which can work for many customers.

Over the years Questionmark has been involved in many standards initiatives, and I thought I’d share in a series of blog articles a personal perspective of some of the key standards that impact assessment.
I’ll start with the standard commonly called AICC or AICC HACP (more formally AICC AGR-10), which is used by learning management systems to call assessment content. Millions of Questionmark assessments are called each year via AICC, and it’s the most successful of all the standards we use.

The AICC is an aviation industry organization founded in 1988, the same year Questionmark. Airlines and airplane makers wanted a way to deliver computerized learning to help people maintain planes that could last the 20 years or so years that the planes themselves would last. The original AICC standard was file based but was soon updated to work over HTTP.

In an assessment context, the AICC standard allows launching and tracking of an assessment:

1.  A Calling Application (for instance an LMS) calls an assessment system saying that it wants to start an assessment.

2.  The assessment system asks the Calling Application for the details of which participant and which assessment.

3.  The Calling Application replies and the assessment starts.

4.  At the end of the assessment, the score is passed back to the Calling Application for tracking.

A key reason the AICC standard is robust and successful is that there is direct server-to-server communication. The two pieces of software communicate directly by HTTPS, and so there is no possibility of disruption or interference by anything at the participant workstation.

I was introduced to the AICC standard in the 90s by Bryan Chapman (now an e-learning analyst) and Questionmark supported it as a way of making it easy for people with LMSs to call Questionmark Perception assessments. We first became certified to the AICC standard in 1999 and we’ve been re-certified several times since. The great thing about the AICC standard is that it really works: Because it’s been tried and trusted over many years, I can think of at least 25 different vendors that Questionmark has interoperated with using this standard, ranging from PeopleSoft and Sun down to much smaller vendors.

My previous post offered three tips on making your assessments more secure and preventing cheating. Here are four more. You will find additional information about this in “Delivering  Assessments Safely and Securely,” and I’ll be mentioning other security tips in my future posts. I hope you will respond with your own ideas about avoiding the problem of cheating on tests.

Screening participants who achieve perfect scores

Given the rarity of achieving perfect scores on assessments, consider doing some investigating when you see perfect scores. Many organizations do this automatically. This might  interview the exam proctor and do other checks to ensure no suspicious behavior has occurred.

Verifying expected IP addresses

If you are administering an assessment  at a specific location, you will likely be able to obtain  the IP address of the computer being used. You can then tell whether participants took the assessment there or at an unauthorized location.

Using Trojan horse or stealth items

Use Trojan horse or stealth items to help detect whether a participant has memorized the answer key. Stealth items look just like the other questions, but they are purposely keyed incorrectly. You can include these items are generally included as non-scored items on the assessment. They will help you detect if a participant is simply memorizing content and keyed correct answers, since they will likely choose alternatives that they have memorized. Participants with overall reasonable assessment scores who got the stealth items “correct” might have memorized the answer key.

Reveal that cheater prevention tactics are used

Informing participants that reviews are regularly conducted to identify cheaters is a simple way to decrease the temptation to cheat. You don’t need to provide details about the sort of reviews you conduct, but do let participants know that cheater-detection tactics are regularly employed.

Posted by Steve Lay

Last month I wrote about our new Community Edition program under which we’ll be releasing selected integration products as free-to-download Community Editions distributed under open source licenses.

Although this program represents an exciting new way to involve the development community, Questionmark is no stranger to working with open source. Moodle is a popular learning management system that is developed by an active community of open source developers. Questionmark has always distributed its connector products for Moodle under a similar open source license, so it is no surprise that our new connector for Moodle 1.9 is available as a Community Edition.

The connector is written in the widely-used PHP system used by Moodle itself and uses QMWISe (Questionmark Web Integrated Services Environment) and PIP (Perception Integration Protocol) to allow Perception tests to be included as activities directly within Moodle courses.

You can find our more information about the connector and the development project behind the Community Edition from our new developer site: developer.questionmark.com. On the site, you’ll find information about our Application Programming Interfaces (APIs) and all of our Community Edition projects. The Community Edition Moodle Connector page on the development site also links to the development project, which is hosted on the popular SourceForge system where you can download the latest version and even browse the code!

And finally, we’ve used Moodle as a development platform for the developer site itself, which means that we can demonstrate the connector working directly from the site. Look for the “Try It Out!” section on the Community Edition Moodle Connector page and see if you can answer my quick quiz on the main features of our Community Edition release program.

The hits keep coming. This month we have improved the look and feel of Questionmark Live with new buttons and improved Question Set sharing. Now when you create one shared question set, you can create another with the same shared permissions. You can also invite someone to join Questionmark Live via a shared question set or stop sharing a particular question set.

These new features are sure to make it easier for you to work collaboratively with your Subject Matter Experts (SMEs), and they’ll help you set up multiple question sets (topics) for item writing workshops. 

Watch the short video below to see how easy it is to create a new question set with sharing already set up!

Last week I wrote about tips for protecting  intellectual property.  It’s equally important keep people from cheating on tests, so here are three pointers on that subject.  I’ll be following this up with more tips in future posts.  Leave me your comments; we can always add to lists like these!

.Screening tests

Consider givinig a small pre-screening test to prevent people from taking an assessment that is beyond their current ability level. If a participant can‘t answer a certain number of these questions correctly they will not be allowed to see the remainder of the assessment. When the time does come for them to take the test,  they will not have already seen its content.

Candidate agreements

Candidate agreements or examination honor codes require  a participant’s agreement before they start an assessment, say by clicking on an “OK “ or  “Yes” button after reading the exam’s code of conduct.

The code might say something like this:  “I agree to answer the questions on this assessment without obtaining assistance from another person or via electronic means. I agree to not to share my answers with anyone during or after the exam. I further agree to not memorize or otherwise steal the intellectual property contained in this exam. I accept that if any of these conditions are violated, my exam results will be set to a zero, I will not be able to retake the exam for a period of 10 years, and I may be charged with a crime under regional laws.”

Here are some topics you might want to cover in a candidate agreement:

• The test vendor will have the option to terminate the assessment if suspicious behavior is detected
• The candidate must abide by the rules of the test center, organization, or program
• The candidate will not provide false ID or false papers
• The candidate cannot take the test on behalf of someone else
• The candidate will not engage in cheating in any form
• The candidate will not help others cheat by disclosing information about the assessment
• The candidate will not use aids that are not allowed
• The candidate will not solicit someone else to take the test
• The candidate will not cause a disturbance in the testing center
• The candidate will not tamper with the test center in any way
• The candidate will not share information

Limiting content exposure/leakage

In order to limit the amount of question content being shown to a participant at any given time, think about using question-by-question templates. These present questions one at a time to participants so that exam content is not completely exposed on screen. Participants who may intend to take pictures of the exam content or otherwise steal intellectual property will not be able to do so all at once.

There are many fine resources for learning how to prevent cheating. Two of thes are books by Dr. Gregory Cizek:  Cheating on Tests: How to Do It, Detect It and Prevent It and Detecting and Preventing Classroom Cheating: Promoting Integrity in Assessment.

There’s also our white paper: “Delivering  Assessments Safely and Securely,” and of course this blog! Watch for more security tips in my future posts.