Standard Setting: An Introduction

greg_pope-150x1502

Posted by Greg Pope

Standard setting was a topic of considerable interest to attendees at the Questionmark 2010 Users Conference  in March.We had some great discussions about standard setting methods and practical applications in some of the sessions I was leading, so I thought I would share some details about this topic here.

Standard setting is generally used in summative criterion referenced contexts. It is the process of setting a “pass/fail” score that distinguishes those participants who have the minimum acceptable level of competence in an area to pass from those participants who do not have the minimum acceptable level of competence in an area. For example, in a crane operation certification course, participants would be expected to have a certain level of knowledge and skills to operate a crane successfully and safely. In addition to a practical test (e.g., operation of a crane in a safe environment) candidates may also be required to take a crane certification exam in which they would need to achieve a certain minimum score in order to be allowed to operate a crane. On the crane certification exam a pass score of 75% or higher is required for a candidate to be able to operate a crane; anything below 75% and they would need to take the course again. Cut scores do not only refer to pass/fail benchmarks. For example, organizations may have several cut scores within an assessment that differentiate between “Advanced”, “Acceptable”, and “Failed” levels.

Cut scores are very common in high and medium-stakes assessment programs; well established processes for setting these cut scores and maintaining them across administrations are available. Generally, one would first build/develop the assessment with the cut score in mind. This would entail selecting questions that represent the proportionate topics areas being covered, ensuring an appropriate distribution of difficulty of the questions, and selecting more questions in the cut score range to maximize the “measurement information” near the cut score.

Once a test form is built it would undergo formal standard setting procedures to set or confirm the cut score(s). Here is a general overview of a typical Modified Angoff type standard setting process:

typical Modified Angoff type standard setting process

Stay tuned for my next post on this topic, in which I will describe some standard setting methods for establishing cut scores.

3 Responses to “Standard Setting: An Introduction”

  1. Silvester says:

    Hi Greg,

    Nice post.

    Can you tell me what source for Angoff procedure you use and is best to communicate with ‘regular’ instructors? I know a few sources, but are interested in yours.

    Kind regards,

    Silvester

  2. Sandra Bourgon says:

    Hello!
    I have a question and was wondering if you could answer it for me. I run a certification process for a health care profession. this is the 2nd year I’ve been involved. Last year I sat through the pass standard setting of the exam and the 250 multiple choice questions were assigned a pass standard using the modified angoff method. This year we had another pass standard setting meeting and my question is as follows:

    If we set the standard for a common question on last years exam, should the same question be reviewed and set again the following year?

    If yes, why?
    If no, why?

    I have been trying to dig around to find an answer and came across your blog and thought I’d give this a try. Thank you for your time…

  3. admin says:

    Hi Sandra,

    I suspect it depends on the consequences of mis-classification and the volume of assessments and how fast the field and the participants are changing. Could I suggest you post this in one of the LinkedIn forums? Perhaps the one for the Institute of Credentialing Excellence, the eLearning Developers Guild or another one where professional test developers will respond. You will probably get several useful responses by doing so.

    Best regards,

    Julie

Leave a Reply