Workshop on Test Development Fundamentals: Q&A with Melissa Fein

Posted by Joan PhaupJoan Phaup 2013 (3)

We will be packing three days of intensive learning and networking into the Questionmark 2014 Users Conference in San Antonio March 4 – 7.

From Bryan Chapman’s keynote on Transforming Open Data into Meaning and Action to case studies, best practice advice, discussions, demos and instruction in the use of Questionmark technologies, there will be plenty to learn!

Even before the conference starts, some delegates will be immersed in pre-conference workshops. This year we’re offering one  full-day workshop and two half-day workshops.

Here’s the line-up:

Today’s conversation is with Melissa Fein, an industrial-organizational psychology consultant and the author of Test Development:  Fundamentals for Certification and Evaluation.

Melissa’s workshop will help participants create effective criterion-reference tests (CRT). It’s designed for people involved in everything from workplace testing and training program evaluation to certifications and academic testing.

What would you say is the most prevalent misconception about CRT?
…that a passing score should be 70 percent. The cutoff for passing might end up to be 70 percent, but that needs to be determined through a standard-setting process. Often people decide on 70 percent because 70 percent is traditional.

What is most important thing to understand about CRT?
It’s crucial to understand how to produce and interpret scores in a way that is fair to all examinees and to those who interpret and use the scores in making decisions, such as hiring people, promoting people, and awarding grades. Scores are imperfect by nature; they will never be perfect. Our goal is to produce quality scores given the limitations that we face.

How does CRT differ in the worlds of workplace testing, training, certification and academic assessment?
The process used to identify testing objectives differs for these different contexts.  However, there are more similarities than differencesin developing CRTs for workplace testing, training, certification and academic assessment.  The principles underlying the construction of quality assessments — such as validity, reliability, and standard setting — don’t differ.

When is CRT the most appropriate choice, as opposed to norm-referenced testing?
Anytime test scores are being compared to a standard, you want to use criterion-referenced testing. With norm referenced tests, you just want to compare one examinee’s scores with another. If you had police officers who have to pass fitness standards — maybe they have to run a mile in a certain amount of time – you would use CRT. But if the officers are running a benefit 5K race, that’s norm-referenced. You just want to find out who comes in first, second and third.

I understand you will be covering testing enigmas during the workshop. What do you have in mind?
Testing enigmas reflect best practices that seem to defy common sense until you look more closely. The biggest enigma occurs in standard setting. When most people think of setting standards for certifications, they like to think of a maximally proficient person. When I ask them to think of a minimally competent person, they think I’m pulling the rug out from under them! But in standard setting, you are trying to determine the difference between passing and failing, so you are looking to identify the minimally competent person: you want to define the line that distinguishes the minimally competent person from someone who is not competent.

What do you hope people will take away from their morning with you?
I hope people will walk away with at least one new idea that they can apply to their testing program. I also hope that they walk away knowing that something they are already doing is a good idea – that the workshop validates something they are doing in their test development work. Sometimes we don’t know why we do certain thing, so it’s good to get some reassurance.


Click here to read a conversation with Rick Ault about Boot Camp. My next post will be a Q&A with item writing workshop instructor Mary Lorenz.

You will save $100 if you register for the conference by January 30th. You can add a workshop to your conference registration or choose your workshop later.

Conference Close-up: Practical Advice on Criterion-Referenced Testing

Posted by Joan Phaup

As we gear up for the Questionmark 2012 Users Conference, we’re especially pleased that there will be two full-day pre-conference workshops on March 20. One will offer basic training for people new to Questionmark and the other will focus on Criterion-Referenced Test Development (CRTD).

Sharon Shrock

Sharon Shrock and Bill Coscarelli, who have delivered keynote addresses at two of our previous conferences, will be leading the CRTD workshop. Having devoted more than 30 years to this subject as university professors, business consultants and authors, Bill and Sharon have a lot of experience and expertise to share.

I can’t wait to see Sharon and Bill in New Orleans, so I did the next best thing by spending a few minutes on the phone with them last week.

Why did you decide to focus on criterion-referenced tests in your work?

Sharon: Initially it was because of instructional design models, in which the role of assessment primarily was to determine if individual learners had mastered the intended objectives.  The question always was, “Can these people do these things?” That argues for measuring against a standard instead of measuring people against one another, which is what happens in norm-referenced testing.

Norm-referenced tests essentially rank people, whereas criterion-referenced tests measure against specific objectives.  From our earliest training in instructional design, we’d go into testing classes that were always about norm referencing: rank-ordering people based on a distribution of scores. There was never any guidance about how to go about testing based on objectives. So we set out to provide advice and instruction and write a book about it.

What do you wish every test author knew about writing effective questions and tests?

Bill: We once wrote a piece called The Two Most Important Things You Can Do to Improve Testing in Your Organization – and what we said there still holds true.

First: write questions above the memorization level. Simply writing questions that require you to regurgitate standard text from a manual is not a predictor of success in the workplace. Second: make sure every test has a meaningful passing score. You should use a solid technique – most likely the Angoff Technique — to set the cut-off score for mastery.  Professionally, ethically, and legally, a cut-off score defining the mastery/non-mastery decision point cannot be defended if it has been chosen in some arbitrary manner.

Bill Coscarelli

Sharon: Those two pieces of advice are in fact the areas where we see the most significant failures to execute adequate assessment on the part of organizations. Both concepts are kind of tricky to grasp — especially the one about writing questions above the memory level.  People understand the idea and agree with the rationale, but they find it very difficult to do. That’s because most of the questions they’ve seen in their lives have been recall level questions. But multiple-choice questions at the recall level aren’t up to the job. Nobody just remembers information on the job! Workers USE information to perform important tasks. We’re not saying memory is bad, but if that’s the only level at which an organization tests, the results won’t predict or improve job performance. Test writers really need to match the assessment to the job.

How will you approach your workshop?

Sharon: We’re well aware that there is so much information out there about testing, and that the people at the workshop are under enormous pressure to produce good tests. So we’ll distill the most useful knowledge. We’ll stay away from heavy duty theoretical stuff and norm-referenced statistics to focus only on material that people can actually use.

Bill: That’s how people can learn a whole lot in a relatively short period of time. We will give them an overview of a professional, doable, and legally defensible model for Criterion-Referenced Testing.  And we’ll give them a 14-step model for developing high-quality tests. The workshop will be a mixture of lecture, discussion, Q & A, and exercises that we’ll work through with participants to illustrate each point.

What will people take away from the day they spend with you in New Orleans?

Sharon: They will be better interpreters of test quality in general.  Even when their kids bring home tests from school, they’ll be able to tell a lot about the test and how relevant it is or is not. More importantly, they’ll understand that creating Criterion-Referenced Tests is, in fact, doable for most people. It takes no more time or resources to test correctly than it takes to test incorrectly. It all comes down to the knowledge and skill of the test writer.

Bill: They’ll know what needs to be done for testing and certification and what the milestones are. And even if they can’t apply everything we teach them right away, they’ll understand their own priorities and see what they can do to make things better right now. They’ll be thinking smart about testing. That’s what we’ll show them how to do.

Early-bird conference registration is open until January 27th. Click here to learn more about the program.

Sharon Shrock & Bill Coscarelli Interview on Criterion-Referenced Testing

Posted by Joan Phaup

I enjoyed talking recently with Sharon Shrock and Bill Coscarelli, who spoke at the Questionmark 2009 Users Conference. Their keynote address covered the 25 years of progress in Criterion-Referenced Test Development and gave everyone at the conference some excellent background on this increasingly important subject.

I had some questions for them about this topic and am happy to share their answers in this podcast.