Cutscores: A book review

Posted By Doug Peterson

You’ve written good items and had them approved. You’ve assembled them into an assessment. Ready to go? Well, you need to determine what constitutes passing and failing — or maybe basic, proficient and advanced skill levels. You can pick a number out of the air, but that’s not necessarily the best or most fair way to do it, especially for high-stakes exams.

Cutscores: A Manual for Setting Standards of Performance on Educational and Occupational Tests, by Michael J. Zieky, Marianne Perie and Samuel A. Livingston, reviews 18 different methods for determining a cutscore, including three different versions of the popular Angoff method. It provides all the information you need to decide which method is best for you and your situation, and then describes how to carry out a cutscore study and document the results.

I thought this book was going to be a tough technical read, but I felt it offered information with which I needed to familiarize myself, so I prepared to slug my way through it. To my surprise, the book is actually very readable. Don’t get me wrong: It’s no exciting adventure novel, but you don’t have to have an advanced degree in education and statistics to understand it.

I also like the way it’s organized. It starts with “What You Need To Know About Cutscores” and moves on to “What You Have To Do Before You Set Cutscores.” Then the authors do something brilliant: rather than go through each method in detail, they take a chapter to describe each method at a high level and explain when you would use it. Then, in the following chapter, they provide detailed information on carrying out each type of cutscore study. This approach makes it easy to determine which method best meets your needs in one chapter, and then focus only on that method in the next chapter.

The book wraps up by explaining what needs to be done after the cutscore study is complete, and includes examples of standard forms such as Nondisclosure Agreements and various evaluation forms in an appendix.

I recommend this book to anyone involved in the development of high-stakes assessments. This book provides the information you need to set up a fair and legally defensible method for determining the appropriate cutscore for your assessment.

Recommended Reading: Learning on Demand by Reuben Tozman

Posted by Jim Farrell

I don’t know about you, but I often feel spoiled by Twitter.

Being busy forces me to mostly consume short articles and blog posts with the attention span similar to my 6-year-old son. Over the course of the year, the pile of books on my nightstand grows, and I fall behind in books I want to read. My favorite thing about this time of the year (besides football and eggnog) is catching up on my reading.

One book that I’ve been really looking forward to reading, since hearing rumors of its creation by the author, is Learning on Demand by Reuben Tozman.

For those of you who are regulars at e-learning conferences, the name Reuben Tozman will not be new to you. Reuben is not one for the status quo. Like many of us, he is constantly looking for the disruptive force that will move the “learner” from the cookie-cutter, one-size-fits-all model that many of us have grown up with to a world where everything revolves around the context of performance. I put the word learner in quotes because Reuben hates the word. We are all learners all of the time in the 70+20+10 world. You are not only a learner when you are logged into your LMS.

Learning on Demand takes the reader through the topics of understanding and designing learning material with the evolving semantic web, the new technologies available today to make learning more effective and efficient, structuring content for an on-demand system, and key skills for instructional designers.

Each chapter includes real-world examples that anyone involved in education will connect with. This isn’t a book that tells you to throw away the baby with the bath water: There are a lot of skills that Instructional Designers use today that will help them be successful in a learning-on-demand world.

Even the appendix of case studies has nuggets to take forward and expand into your everyday work. My favorite was a short piece on work Reuben did with the Forum for International Trade Training (FITT). They called it a “J3 vision” which goes beyond training to performance support. The “Js” are:  J1 – just enough, J2 – Just in time (regardless of time and/or location), and J3 – Just for me (delivered in the medium I like to learn in,) (Notice I did not say learning style: That is a discussion for another time.) To me, this is the perfect way to define good performance support.

I think it would be good for Instructional Designers to put their Dick and Carey books into the closet and keep Reuben’s book close at hand.