Posted by John Kleeman
How many times is it fair and proper for a participant to retake an assessment if they fail?
One of our customers asked me about this recently in regard to a certification exam. I did some research and thought I’d share it here.
For a few kinds of assessments, you would normally only allow a single attempt, typically if you are measuring something at a specific point in time. A pre-course or post-course test might only be useful if it is taken right before or right after a training course.
Regarding assessments that just give retrieval practice or reinforce learning, you needn’t be concerned. It may be fine to allow as many retakes as people want. The more times they practice answering the questions, the more they will retain the learning.
But how can you decide how many attempts to allow at a certification assessment measuring competence and mastery?
Consider test security
Retakes can jeopardize test security. Someone might take and retake a test to harvest the items to share with others. The more retakes allowed, the more this risk increases.
International Test Commission draft security guidelines say:
“Retake policies should be developed to reduce the opportunities for item harvesting and other forms of test fraud. For example, a test taker should not be allowed to retake a test that he or she “passed” or retake a test until a set amount of time has passed.”
Consider measurement error
All assessment scores have measurement error. A certification exam classifies people as having mastery (pass) or not (failing), but it doesn’t do so perfectly.
If you allow repeat retakes, you increase the risk of classifying someone as a master who is not competent, but you also decrease the risk of classifying a competent person as having failed. This is because someone can suffer test anxiety or be ill or make a stupid mistake and fail the test despite being competent.
Require participants to wait for retakes
It’s usual to require a time period to elapse before a retake. This stops people from using quick, repeated retakes to take unfair advantage of measurement error. It also encourage reflection and re-learning before the next attempt. Standard 13.6 in the Standards for Educational and Psychological Testing says:
“students. . . should have a reasonable number of opportunities to succeed. . . the time intervals between the opportunities should allow for students to have the opportunity to obtain the relevant instructional experiences.”
If we had a perfectly reliable assessment, there would be no concern about multiple attempts. Picking the number of attempts is a compromise between what is fair to the participants and the limitations of our resources as assessment developers.
Think about test preparation
Could your retake policy affect how people prepare for the exam?
If retakes are easily available, some participants might prepare less effectively, hoping that they can “wing it” since they can retake at will. On the other hand, if retakes are limited, this could increase test anxiety and stress. It could also increase the motivation to cheat.
What about fairness?
Some people suffer test anxiety, some people make silly mistakes on the test or use poor time management, and some may be not at their full capacity on the day of the exam. It’s usually fair to offer a retake in such situations. If you do not offer sufficient opportunities to retake, this will impact the face validity of the assessment: people might not consider it fair.
If your exam is open the public, you may not be able to limit retakes. Imagine a country where you were not allowed to retake your driving test once you’d failed it 3 times! It might make the roads safer, but most people wouldn’t see it as equitable.
In my next post on this subject, I will share what some organizations do in practice and offer some steps for arriving at an answer that will be suitable for your organization.