Posted by Greg Pope
Standard setting was a topic of considerable interest to attendees at the Questionmark 2010 Users Conference in March.We had some great discussions about standard setting methods and practical applications in some of the sessions I was leading, so I thought I would share some details about this topic here.
Standard setting is generally used in summative criterion referenced contexts. It is the process of setting a “pass/fail” score that distinguishes those participants who have the minimum acceptable level of competence in an area to pass from those participants who do not have the minimum acceptable level of competence in an area. For example, in a crane operation certification course, participants would be expected to have a certain level of knowledge and skills to operate a crane successfully and safely. In addition to a practical test (e.g., operation of a crane in a safe environment) candidates may also be required to take a crane certification exam in which they would need to achieve a certain minimum score in order to be allowed to operate a crane. On the crane certification exam a pass score of 75% or higher is required for a candidate to be able to operate a crane; anything below 75% and they would need to take the course again. Cut scores do not only refer to pass/fail benchmarks. For example, organizations may have several cut scores within an assessment that differentiate between “Advanced”, “Acceptable”, and “Failed” levels.
Cut scores are very common in high and medium-stakes assessment programs; well established processes for setting these cut scores and maintaining them across administrations are available. Generally, one would first build/develop the assessment with the cut score in mind. This would entail selecting questions that represent the proportionate topics areas being covered, ensuring an appropriate distribution of difficulty of the questions, and selecting more questions in the cut score range to maximize the “measurement information” near the cut score.
Once a test form is built it would undergo formal standard setting procedures to set or confirm the cut score(s). Here is a general overview of a typical Modified Angoff type standard setting process:
Stay tuned for my next post on this topic, in which I will describe some standard setting methods for establishing cut scores.