5 Ways to Limit the Use of Breached Assessment Content

Austin Fossey-42Posted by Austin Fossey

In an earlier post, Questionmark’s Julie Delazyn listed 11 tips to help prevent cheating. The third item on that list related to minimizing item exposure; i.e., limiting how and when people can see an item so that content will not be leaked and used for dishonest purposes.

During a co-presentation with Manny Straehle of Assessment, Education, and Research Experts at a Certification Network Group quarterly meeting, I presented a set of considerations that can affect the severity of item exposure. My message was that although item exposure may not be a problem for some assessment programs, assessment managers should consider the design, purpose, candidate population, and level of investment for their assessment when evaluating their content security requirements.

mitigating risk

If item exposure is a concern for your assessment program, there are two ways to mitigate the effects of leaked content: limiting opportunities to use the content, and identifying the breach so that it can be corrected. In this post, I will focus on ways to limit content-using opportunities:

Multiple Forms

Using different assessment forms lowers the number of participants who will see an item in delivery. Having multiple forms also lowers the probability that someone with access to a breached item will actually get to put that information to use. Many organizations achieve this by using multiple, equated forms which are systematically assigned to participants to limit joint cheating or to limit item exposure across multiple retakes. Some organizations also achieve this through the use of randomly generated forms like those in Linear-on-the-Fly Testing (LOFT) or empirically generated forms like those in Computer Adaptive Testing (CAT).

Frequent Republishing

Assessment forms are often cycled in and out of production on a set schedule. Decreasing the amount of time a form is in production will limit the impact of item exposure, but it also requires more content and staff resources to keep rotating forms.

Large Item Banks

Having a lot of items can help you make lots of assessment forms, but this is also important for limiting item exposure in LOFT or CAT. Item banks can also be rotated. For example, some assessment programs will use an item bank for particular testing windows or geographic regions and then switch them at the next administration.

Exposure Limits

If your item bank can support it, you may also want to put an exposure limit on items or assessment forms. For example, you might set up a rule where an assessment form remains in production until it has been delivered 5,000 times. After that, you may permanently retire that form or shelve it for a predetermined period and use it again later. An extreme example would be an assessment program that only delivers an item during a single testing window before retiring it. The limit will depend on your risk tolerance, the number of items you have available, and the number of participants taking the assessment. Exposure limits are especially important in CAT where some items will get delivered much more frequently than others due to the item selection algorithm.

Short Testing Windows

When participants are only allowed to take a test during a short time period, there are fewer opportunities for people to talk about or share content before the testing window closes. Short testing windows may be less convenient for your participant population, but you can take advantage of the extra downtime to spend time detecting item breaches, developing new content, and performing assessment maintenance.

In my next post, I will provide an overview of methods for identifying instances of an item breach.

Going Green at the Questionmark Users Conference

Joan Phaup

Posted by Joan Phaup

As part of our efforts to make the Questionmark  Users  Conference agreener event than ever, we’re pleased to have selected a hotel that has been widely recognized for its green initiatives.

recycle

The Hilton Miami Downtown has demonstrated its commitment to preserving resources through its participation in the Florida Green Lodging Program. The program, established in 2004 by the Florida Department of Environmental Protection (DEP) with the intent of recognizing and rewarding environmentally conscientious lodging facilities in the state, is helping Florida’s lodging industry conserve and protect the state’s natural resources.

As a Florida Green Lodging Program designated facility, the hotel has formed a “Green Team” that focuses on ways to reduce impact on  natural resources. The Green Team’s efforts so far have focused on water conservation, energy efficiency, waste reduction and clean air practices. The hotel has now uses low-flow shower heads and toilets, digital programmable thermostats, energy-efficient lighting and recycled paper products. It also utilizes  bulk purchasing and has in-room and vending area recyling bins.  There’s a comprehensive recycling program for bulbs, cardboard, paper, plastic and toner cartridges. Clean air measures include the use of high efficiency air filters and the use of environmentally sound cleaning products.

HiltonMiamiDowntown_lg

We’re also “greening” the conference by reducing the amount of paper we use, decreasing the size of our conference notebooks and  making handouts available online.  In addition, we’ll reduce our use of bottled water, create re-usable signage and provide recycling bins for badge holders, lanyards and, of course, paper!

With the excellent program we have planned for Questionmark users, we expect the conference to inspire participants with new, spring-green shoots for their assessment programs. See you at the conference March 14 – 17! Our $100 earlybird registration ends January 22nd, so register soon!