Conference Close-up: Practical Advice on Criterion-Referenced Testing

Posted by Joan Phaup

As we gear up for the Questionmark 2012 Users Conference, we’re especially pleased that there will be two full-day pre-conference workshops on March 20. One will offer basic training for people new to Questionmark and the other will focus on Criterion-Referenced Test Development (CRTD).

Sharon Shrock

Sharon Shrock and Bill Coscarelli, who have delivered keynote addresses at two of our previous conferences, will be leading the CRTD workshop. Having devoted more than 30 years to this subject as university professors, business consultants and authors, Bill and Sharon have a lot of experience and expertise to share.

I can’t wait to see Sharon and Bill in New Orleans, so I did the next best thing by spending a few minutes on the phone with them last week.

Why did you decide to focus on criterion-referenced tests in your work?

Sharon: Initially it was because of instructional design models, in which the role of assessment primarily was to determine if individual learners had mastered the intended objectives.  The question always was, “Can these people do these things?” That argues for measuring against a standard instead of measuring people against one another, which is what happens in norm-referenced testing.

Norm-referenced tests essentially rank people, whereas criterion-referenced tests measure against specific objectives.  From our earliest training in instructional design, we’d go into testing classes that were always about norm referencing: rank-ordering people based on a distribution of scores. There was never any guidance about how to go about testing based on objectives. So we set out to provide advice and instruction and write a book about it.

What do you wish every test author knew about writing effective questions and tests?

Bill: We once wrote a piece called The Two Most Important Things You Can Do to Improve Testing in Your Organization – and what we said there still holds true.

First: write questions above the memorization level. Simply writing questions that require you to regurgitate standard text from a manual is not a predictor of success in the workplace. Second: make sure every test has a meaningful passing score. You should use a solid technique – most likely the Angoff Technique — to set the cut-off score for mastery.  Professionally, ethically, and legally, a cut-off score defining the mastery/non-mastery decision point cannot be defended if it has been chosen in some arbitrary manner.

Bill Coscarelli

Sharon: Those two pieces of advice are in fact the areas where we see the most significant failures to execute adequate assessment on the part of organizations. Both concepts are kind of tricky to grasp — especially the one about writing questions above the memory level.  People understand the idea and agree with the rationale, but they find it very difficult to do. That’s because most of the questions they’ve seen in their lives have been recall level questions. But multiple-choice questions at the recall level aren’t up to the job. Nobody just remembers information on the job! Workers USE information to perform important tasks. We’re not saying memory is bad, but if that’s the only level at which an organization tests, the results won’t predict or improve job performance. Test writers really need to match the assessment to the job.

How will you approach your workshop?

Sharon: We’re well aware that there is so much information out there about testing, and that the people at the workshop are under enormous pressure to produce good tests. So we’ll distill the most useful knowledge. We’ll stay away from heavy duty theoretical stuff and norm-referenced statistics to focus only on material that people can actually use.

Bill: That’s how people can learn a whole lot in a relatively short period of time. We will give them an overview of a professional, doable, and legally defensible model for Criterion-Referenced Testing.  And we’ll give them a 14-step model for developing high-quality tests. The workshop will be a mixture of lecture, discussion, Q & A, and exercises that we’ll work through with participants to illustrate each point.

What will people take away from the day they spend with you in New Orleans?

Sharon: They will be better interpreters of test quality in general.  Even when their kids bring home tests from school, they’ll be able to tell a lot about the test and how relevant it is or is not. More importantly, they’ll understand that creating Criterion-Referenced Tests is, in fact, doable for most people. It takes no more time or resources to test correctly than it takes to test incorrectly. It all comes down to the knowledge and skill of the test writer.

Bill: They’ll know what needs to be done for testing and certification and what the milestones are. And even if they can’t apply everything we teach them right away, they’ll understand their own priorities and see what they can do to make things better right now. They’ll be thinking smart about testing. That’s what we’ll show them how to do.

Early-bird conference registration is open until January 27th. Click here to learn more about the program.

Leave a Reply