Workshop on Test Development Fundamentals: Q&A with Melissa Fein

Posted by Joan PhaupJoan Phaup 2013 (3)

We will be packing three days of intensive learning and networking into the Questionmark 2014 Users Conference in San Antonio March 4 – 7.

From Bryan Chapman’s keynote on Transforming Open Data into Meaning and Action to case studies, best practice advice, discussions, demos and instruction in the use of Questionmark technologies, there will be plenty to learn!

Even before the conference starts, some delegates will be immersed in pre-conference workshops. This year we’re offering one  full-day workshop and two half-day workshops.

Here’s the line-up:

Today’s conversation is with Melissa Fein, an industrial-organizational psychology consultant and the author of Test Development:  Fundamentals for Certification and Evaluation.

Melissa’s workshop will help participants create effective criterion-reference tests (CRT). It’s designed for people involved in everything from workplace testing and training program evaluation to certifications and academic testing.

What would you say is the most prevalent misconception about CRT?
…that a passing score should be 70 percent. The cutoff for passing might end up to be 70 percent, but that needs to be determined through a standard-setting process. Often people decide on 70 percent because 70 percent is traditional.

What is most important thing to understand about CRT?
It’s crucial to understand how to produce and interpret scores in a way that is fair to all examinees and to those who interpret and use the scores in making decisions, such as hiring people, promoting people, and awarding grades. Scores are imperfect by nature; they will never be perfect. Our goal is to produce quality scores given the limitations that we face.

How does CRT differ in the worlds of workplace testing, training, certification and academic assessment?
The process used to identify testing objectives differs for these different contexts.  However, there are more similarities than differencesin developing CRTs for workplace testing, training, certification and academic assessment.  The principles underlying the construction of quality assessments — such as validity, reliability, and standard setting — don’t differ.

When is CRT the most appropriate choice, as opposed to norm-referenced testing?
Anytime test scores are being compared to a standard, you want to use criterion-referenced testing. With norm referenced tests, you just want to compare one examinee’s scores with another. If you had police officers who have to pass fitness standards — maybe they have to run a mile in a certain amount of time – you would use CRT. But if the officers are running a benefit 5K race, that’s norm-referenced. You just want to find out who comes in first, second and third.

I understand you will be covering testing enigmas during the workshop. What do you have in mind?
Testing enigmas reflect best practices that seem to defy common sense until you look more closely. The biggest enigma occurs in standard setting. When most people think of setting standards for certifications, they like to think of a maximally proficient person. When I ask them to think of a minimally competent person, they think I’m pulling the rug out from under them! But in standard setting, you are trying to determine the difference between passing and failing, so you are looking to identify the minimally competent person: you want to define the line that distinguishes the minimally competent person from someone who is not competent.

What do you hope people will take away from their morning with you?
I hope people will walk away with at least one new idea that they can apply to their testing program. I also hope that they walk away knowing that something they are already doing is a good idea – that the workshop validates something they are doing in their test development work. Sometimes we don’t know why we do certain thing, so it’s good to get some reassurance.


Click here to read a conversation with Rick Ault about Boot Camp. My next post will be a Q&A with item writing workshop instructor Mary Lorenz.

You will save $100 if you register for the conference by January 30th. You can add a workshop to your conference registration or choose your workshop later.

Leave a Reply