Our first Beyond Multiple Choice: Nine Ways to Leverage Technology for Better Assessments Web seminar met with an enthusiastic response: it filled up almost instantly! So we are offering the same seminar again on Wednesday, December 16th, at 3 p.m. Eastern Time.

Assessments play a vital role in measuring people’s knowledge, skills and attitudes. They also help organizations improve performance, manage workforce competencies, and ensure regulatory compliance. How can you create and assessments that produce appropriate, actionable results? What can you do to ensure the quality of questions and the security of assessments all the way from authoring and scheduling to administration, reporting, and analysis? How can you make the best use of online authoring, reporting, analytical, and security tools? These and many other questions will be addressed in this free, hour-long session, which will include opportunities for you to ask questions of your own.

We welcome you to learn more about this Webinar and register online.

This is the second of a series of blog posts on assessment standards. Today I’d like to focus on the IMS QTI (Question and Test Interoperability) Specification.

It’s worth mentioning the difference between Specifications and Standards: Specifications are documents that industry bodies have agreed on (like IMS QTI XML), while Standards have been published and committed to by a formal legal body (like AICC or HTML). A Specification is less formal than a Standard but still can be very useful for interoperability.

Questionmark was one of the originators of QTI. When we migrated our assessment platform from Windows to the Web in the 1990s, our customers had to migrate their questions from one platform to the other. As you will know, it takes a lot of time to write high quality questions, and so it’s important to be able to carry them forward independently of technology. We knew that we’d be improving our software over the years and we wanted to ensure the easy transfer of questions from one version to the next. So we came up with QML (Question Markup Language), an open and platform-independent method of maintaining questions that makes it easy for customers to move forward in the future.

Although QML did solve the problem of moving questions between Questionmark versions, we met many customers who had difficulty bringing content created in another vendor’s proprietary format  into Questionmark. We  wanted to help them, and we also wanted to embrace openness and allow Questionmark customers to export out their questions in a standard format if they ever wanted to leave us. So we worked with other vendors within the umbrella of the IMS Global Learning Consortium to come up with QTI XML, a language that describes questions in a technology-neutral way.  I was involved in the work defining IMS QTI as were several of my colleagues: Paul Roberts did a lot of technical design, Eric Shepherd led the IMS working group that made QTI version 1, and Steve Lay (before joining Questionmark) led the version 2 project.

Here is a fragment of QTI XML and you can see that it is a just-about-human-readable way of describing a question.

My previous post offered four tips on making your assessments more secure and preventing cheating.  Aside from verifying IP addresses and running a Trojan horse or stealth items to help detect whether a participant has memorized the answer key, there are some physical actions you can take to avoid the problem and reduce the temptation to cheat.

Proper seating arrangements for participants

Seating participants with adequate space between them and giving them limited ability to see another participant‘s screen or paper are important strategies for enhancing test security. The proctor should be aware of cheating techniques such as the ―”flying V” seating arrangement where the “giver” at the point of the V feeds information to a number of “receivers” behind them. The givers and receivers can communicate in a number of ways, using sign language, dropping notes on the floor, etc. (Dr. Gregory Cizek’s book “Cheating on Tests: How to Do it, Detect it, and Prevent it,” will tell you  more about this and other aspects of cheating.)

Example of the “flying V” answer copying formation (Cizek, 1999):

Using unique make-up exams

Many organizations offer make-up exams for participants who were sick or had legitimate excuses for not being able to take an assessment at the scheduled date and time. If you use the same exam that was administered at the scheduled date and time for their make-up exam, you open yourself to risks of the exam form being compromised. Sometimes the make-up exams are not administered in the same strict proctored environment as the scheduled exam, allowing participants the opportunity to cheat or steal content.

Using more constructed response questions

Constructed response questions, like essay or short answer questions, provide less opportunity for participants to cheat because they require them to produce unique answers to questions. There is no answer key to steal, and participants who copied other people’s constructed response answers are easily identified via a side-by-side comparison of answers.

I hope you enjoyed this three part series on preventing cheating.  You will find more information about  various means for deploying many different types of assessments in our white paper, “Delivering  Assessments Safely and Securely.”

My previous post offered three tips on making your assessments more secure and preventing cheating. Here are four more. You will find additional information about this in “Delivering  Assessments Safely and Securely,” and I’ll be mentioning other security tips in my future posts. I hope you will respond with your own ideas about avoiding the problem of cheating on tests.

Screening participants who achieve perfect scores

Given the rarity of achieving perfect scores on assessments, consider doing some investigating when you see perfect scores. Many organizations do this automatically. This might  interview the exam proctor and do other checks to ensure no suspicious behavior has occurred.

Verifying expected IP addresses

If you are administering an assessment  at a specific location, you will likely be able to obtain  the IP address of the computer being used. You can then tell whether participants took the assessment there or at an unauthorized location.

Using Trojan horse or stealth items

Use Trojan horse or stealth items to help detect whether a participant has memorized the answer key. Stealth items look just like the other questions, but they are purposely keyed incorrectly. You can include these items are generally included as non-scored items on the assessment. They will help you detect if a participant is simply memorizing content and keyed correct answers, since they will likely choose alternatives that they have memorized. Participants with overall reasonable assessment scores who got the stealth items “correct” might have memorized the answer key.

Reveal that cheater prevention tactics are used

Informing participants that reviews are regularly conducted to identify cheaters is a simple way to decrease the temptation to cheat. You don’t need to provide details about the sort of reviews you conduct, but do let participants know that cheater-detection tactics are regularly employed.

Last week I wrote about tips for protecting  intellectual property.  It’s equally important keep people from cheating on tests, so here are three pointers on that subject.  I’ll be following this up with more tips in future posts.  Leave me your comments; we can always add to lists like these!

.Screening tests

Consider givinig a small pre-screening test to prevent people from taking an assessment that is beyond their current ability level. If a participant can‘t answer a certain number of these questions correctly they will not be allowed to see the remainder of the assessment. When the time does come for them to take the test,  they will not have already seen its content.

Candidate agreements

Candidate agreements or examination honor codes require  a participant’s agreement before they start an assessment, say by clicking on an “OK “ or  “Yes” button after reading the exam’s code of conduct.

The code might say something like this:  “I agree to answer the questions on this assessment without obtaining assistance from another person or via electronic means. I agree to not to share my answers with anyone during or after the exam. I further agree to not memorize or otherwise steal the intellectual property contained in this exam. I accept that if any of these conditions are violated, my exam results will be set to a zero, I will not be able to retake the exam for a period of 10 years, and I may be charged with a crime under regional laws.”

Here are some topics you might want to cover in a candidate agreement:

• The test vendor will have the option to terminate the assessment if suspicious behavior is detected
• The candidate must abide by the rules of the test center, organization, or program
• The candidate will not provide false ID or false papers
• The candidate cannot take the test on behalf of someone else
• The candidate will not engage in cheating in any form
• The candidate will not help others cheat by disclosing information about the assessment
• The candidate will not use aids that are not allowed
• The candidate will not solicit someone else to take the test
• The candidate will not cause a disturbance in the testing center
• The candidate will not tamper with the test center in any way
• The candidate will not share information

Limiting content exposure/leakage

In order to limit the amount of question content being shown to a participant at any given time, think about using question-by-question templates. These present questions one at a time to participants so that exam content is not completely exposed on screen. Participants who may intend to take pictures of the exam content or otherwise steal intellectual property will not be able to do so all at once.

There are many fine resources for learning how to prevent cheating. Two of thes are books by Dr. Gregory Cizek:  Cheating on Tests: How to Do It, Detect It and Prevent It and Detecting and Preventing Classroom Cheating: Promoting Integrity in Assessment.

There’s also our white paper: “Delivering  Assessments Safely and Securely,” and of course this blog! Watch for more security tips in my future posts.

Protecting the intellectual property contained in a test or exam is essential, not only because of the time, effort and cost of creating assessments but also because IP theft undermines the accurate measurement of knowledge and skills.

Protecting intellectual property protects the credibility of tests. Here are four tips for helping to ensure the security of intellectual property:

Create and administer multiple test forms

Rather than having only one form of the assessment being administered, delivering multiple forms of the same exam can help limit item exposure. This method also allows for the possibility of interspersing large-scale integrated beta test questions within the forms to collect psychometric information on newly developed questions.

Restrict and control administration of beta test items

Beta testing questions is an important part of high-stakes assessment, ensuring the psychometric quality of questions before they appear on actual assessments. However, it is vital that a well conceptualized beta test model is in effect to limit the exposure of newly developed questions to participants.

Update exam forms periodically

Letting exam forms become stale can over-expose questions to participants, increasing the likelihood of IP theft. An organization could consider retiring old exam forms and turning them into exam prep materials that can be sold to participants. In this way, participants could periodically expect new practice questions.

Produce exam prep materials

Organizations should consider making exam prep materials available to participants before an assessment. This will help reduce the demand for participants to try to obtain exam questions via illegal means as they will have access to the type of questions that will be asked on the actual assessment.

For more details on this subject, plust information about various means for deploying a wide range of assessment types with assurance, download our White Paper: Delivering Assessments Safely and Securely.

Assessments provide a valuable tool for helping organizations properly design effective and useful learning experiences. Doing so involves a five-step process.

Step 1: Define the objectives. An objective might be to increase customer satisfaction, reduce error rates or improve safety.

Step 2: Ask what knowledge and skills are required to meet the objectives. In a college or university course on organic chemistry, for example, it’s important to ask what are the knowledge and skills required to comprehend those concepts and use them. The answers to that question will help the professor establish the topic structure to define the knowledge, skills, abilities and attitudes required to meet the objectives.

Step 3: Run a needs analysis survey or skills gap survey. Here, people take a needs assessment to reveal the knowledge and skills they already have as well as what they still need. A gap analysis can be derived from the difference between what is required and people’s current knowledge and skills.

Step 4: Develop a learning plan. That plan will describe the learning objectives and explain how the plan will be administered. The learning objectives will guide the production of learning materials and assessments. Facilitating the learning process might involve instructor-led training, coaching by managers, or e-learning courses.

Step 5: Conduct a pre-learning assessment. The pre-learning assessment will have two purposes: to create intrigue about the course and  to guide each participant to the right learning experience. Advanced and novice learners will require different approaches.

To learn more about  how instructors and organizations can use assessments to improve learning, download the white paper Assessments through the Learning Process.

Posted by Greg Pope

I had a great time putting together an eight-part series on Item Analysis Analytics for this blog and was pleased with the interest it received.

When a reader asked if it would be possible to present all the posts in a single document I thought hey, let’s present the content of these articles in the form of a Questionmark White Paper! So here it is for you to download with our compliments.

I hope the paper helps you in your efforts to create test questions that make the grade!

In talking with some of our customers last week, I was reminded how valuable it can be to offer participants topic based feedback.

Obviously, everyone wants to know whether they’ve passed or failed a test. And most people look at their feedback on questions they got wrong, to understand how to improve. But you can get a single question wrong for many different reasons including misunderstanding the question, making a mistake or having a tiny gap in knowledge. However, if you score weakly in a topic area, it very likely means you have a weakness in that area that needs addressing.

In many ways, topic feedback is the Cinderella in the feedback world. Everyone expects assessment level feedback and item level feedback (perhaps it’s unfair to call them ugly sisters because they are useful and valuable), but there is a huge and often untapped learning value in topic feedback. For pre tests, post course tests, quizzes during learning and practice tests particularly, topic feedback is vital.

Suppose someone is taking an assessment in health and safety and they score 66% and this is a passing score. Sounds good! But what happens if, as in the screenshot below, they’ve scored very well in some topics and poorly in others?

In this example, the fact that someone is very weak in electrical safety could well be concerning. (Don’t let them set up the lighting for the Holidays Ball!)

It’s obvious that you want to give people feedback on the topic level, but in many tools this isn’t as easy to set up as it should be. Questionmark Perception can be a fairy godmother for topic based feedback. There are lots of easy-to-use capabilities to present topic based feedback. Here are some links to support resources to help you create topic feedback in Perception.

• You can easily create topic outcomes with feedback for different topic scores in Authoring Manager
• You set these as standard at the topic level, but can override to adjust on a per assessment level
• You can also make a topic a prerequisite for passing an assessment (for instance to prevent someone passing an assessment unless they get 60% or a specified score in key topics).
• If you want to display only some topics in the list and not all, for instance if some of your topics aren’t meaningful to the participant, you can define which topics are reported on.
• And then the feedback is displayed to participants at the end of assessment easily as in the screenshot above.

I hope you find this useful in getting your topic feedback working and helping your learners achieve their full potential.