How can a randomized test be fair to all?

Joan Phaup 2013 (3) Posted by Joan Phaup

James Parry, who is test development manager at the U.S Coast Guard Training Center in Yorktown, Virginia, will answer this question during a case study presentation the Questionmark Users Conference in San Antonio March 4 – 7. He’ll be co-presenting with LT Carlos Schwarzbauer, IT Lead at the USCG Force Readiness Command’s Advanced Distributed Learning Branch.

James and I spoke the other day about why tests created from randomly drawn items can be useful in some cases—but also about their potential pitfalls and some techniques for avoiding them.

When are randomly designed tests an appropriate choice?

James Parry

James Parry

There are several reasons to use randomized tests.  Randomization is appropriate when you think there’s a possibility of participants sharing the contents of their test with others who have not taken it.  Another reason would be in a computer lab style testing environment where you are testing many on the same subject at the same time with no blinders between the computers. So even if participants look at the screens next to them, chances are they won’t see the same items.

How are you using randomly designed tests?

We use randomly generated tests at all three levels of testing low-, medium- and high-stakes.  The low- and medium-stakes tests are used primarily at the schoolhouse level for knowledge- and performance-based knowledge quizzes and tests.  We are also generating randomized tests for on-site testing using tablet computers or local installed workstations.

Our most critical use is for our high-stakes enlisted advancement tests, which are administered both on paper and by computer. Participants are permitted to retake this test every 21 days if they do not achieve a passing score.  Before we were able to randomize the test there were only three parallel paper versions. Candidates knew this so some would “test sample” without studying to get an idea of every possible question. They would retake the first version, then the second, and so forth until they passed it. With randomization the word has gotten out that this is not possible anymore.

What are the pitfalls of drawing items randomly from an item bank?

The biggest pitfall is the potential for producing tests that have different levels of difficulty or that don’t present a balance of questions on all the subjects you want to cover. A completely random test can be unfair.  Suppose you produce a 50-item randomized test from an entire test item bank of 500 items.   Participant “A” might get an easy test, “B” might get a difficult test and “C” might get a test with 40 items on one topic and 10 on the rest and so on.

How do you equalize the difficulty levels of your questions?

This is a multi-step process. The item author has to make sure they develop sufficient numbers of items in each topic that will provide at least 3 to 5 items for each enabling objective.  They have to think outside the box to produce items at several cognitive levels to ensure there will be a variety of possible levels of difficulty. This is the hardest part for them because most are not trained test writers.

Once the items are developed, edited, and approved in workflow, we set up an Angoff rating session to assign a cut score for the entire bank of test items.  Based upon the Angoff score, each item is assigned a difficulty level of easy, moderate or hard and assigned a metatag to match within Questionmark.  We use a spreadsheet to calculate the number and percentage of available items at each level of difficulty in each topic. Based upon the results, the spreadsheet tells how many items to select from the database at each difficulty level and from each topic. The test is then designed to match these numbers so that each time it is administered it will be parallel, with the same level of difficulty and the same cut score.

Is there anything audience members should do to prepare for this session?

Come with an open mind and a willingness to think outside of the box.

How will your session help audience members ensure their randomized tests are fair?

I will give them the tools to use starting with a quick review of using the Angoff method to set a cut score and then discuss the inner workings of the spreadsheet that I developed to ensure each test is fair and equal.

***

See more details about the conference program here and register soon.

Barcelona or San Antonio or both?

John Kleeman HeadshotPosted by John Kleeman

Questionmark users conferences are unforgettable. I’ve been to all 14 of them so far and each is engraved in my memory as an empowering, enriching and mesmerising event.  We are running two user conferences in the next few months and if you have a chance to attend one (or both!) I promise you won’t regret it.barcelona

Our first upcoming conference is the Questionmark European conference in Barcelona, Spain on 10-12 November. Barcelona is one of the most exciting cities in Europe and will be a great place to learn from other assessment professionals. You can see details at www.questionmark.com/uk/conference.River Walk San Antonio

Our second upcoming conference is the Questionmark US User Conference in San Antonio, Texas on 4-7 March, 2014. San Antonio is the home of the Alamo and the conference venue is part of the River Walk — a uniquely peaceful and positive environment for a conference. You can see details at www.questionmark.com/us/conference.

Here are five reasons I think Questionmark conferences are worth coming to:

1. Learn about assessments. I’ve been working with assessments for over 25 years … I know a lot, but  I’m still learning. Quizzes, surveys, tests and exams are hugely powerful ways of measuring human behavior and helping organizations improve. There is so much to learn.

2. Learn from Questionmark.  Our best presenters and technical experts are at the conference, and they have a lot to share.

Conversation at a Questionmark user conference3. Learn from peers. Most attendees say that the best thing about a Questionmark user conference is that they meet and learn from peers who have similar issues to them. A problem shared is often a problem solved, and you can find out what other people have done in their organization to solve the problems you are facing in yours.

4. Influence the future of the product. What we learn at these conferences contributes hugely to how we improve our products and services. Our product owners (people like Jim Farrell, Austin Fossey, Doug Peterson and Steve Lay)  attend the conferences and listen carefully to what our customers say.

5. Great cities. We know that people who come to our conferences go back to their organization passionate about online assessments and enthusiastic about wider use of Questionmark. We choose great venues for our conferences, and provide memorable experiences in a great environment so the conferences are fantastic personal experiences as well as being fulfilling learning opportunities.

I look forward to meeting readers of this blog at the conferences. And if any conference attendee can name the cities where the 14 conferences prior to these ones were held, I will buy you to a drink of your choice!

Get details here for the European Conference and here for the US Conference.

Planning for a great conference in Baltimore next March

Posted by Joan Phaup

During what may seem like the lazy, hazy, crazy days of summer we are  busy planning the program for the Questionmark 2013 Users Conference in Baltimore March 3 – 6.

We are thrilled about the location of this conference, near the busy harbor in a city that’s easy to reach–  and where there is plenty to see and do.  A top priority is planning the conference program, which will include sessions on best practices, the use of Questionmark features and functions, demos of the latest technologies, case studies and peer discussions.

Baltimore's inner harbor

We provide much of this content ourselves, but customers enrich the program by presenting case studies and leading discussions. We are actively seeking ideas for these sessions, so Questionmark users are invited to check out the conference call for proposals and respond by September 14.

We know the conference seems a long way off and understand that you may not be sure of your plans yet, but we would like to hear from you regardless.

  • Do you have a success story to share about your use of Questionmark assessments?
  • Have you had experiences or learned lessons that would be helpful to others?
  • Is there a topic you’d like to talk about with fellow learning and assessment professionals?

If you can answer “yes” to any of these questions, we would welcome your ideas!

Conference goers invariably say that one of the best things about this event is the opportunity to learn from fellow Questionmark users and to find out how different organizations use assessments. So we hope you will spend a some time pondering what you would like to share and letting us know about it!