How many test or exam retakes should you allow? Part 2

John Kleeman HeadshotPosted by John Kleeman

In my last post, I offered some ideas about what to consider when determining your retake policy regarding a certification assessment measuring competence and mastery. Some of the issues to balance are test security, fairness, a delay between retakes and the impact of retakes on test preparation. In this conclusion to the post, I’ll share what a few other organizations do and how you might approach deciding the number of retakes to allow.

Here is how a few respected certification programmes manage retakes

SAP have the following rules in their certification programme:

No candidate may participate in the same examination for the same release more than three times. A candidate who has failed at an examination three times for a release may not attempt that examination again until the next release. 

Microsoft allow up to 5 attempts in a 12-month period and then impose a 12-month waiting period. They also have gaps of several days between retakes, with the number of days increasing for subsequent retakes.

The US financial regulator FINRA requires a waiting time of 30 days between exams, but if you fail an exam three or more times in succession, you must wait 6 months before taking it again.

What’s the right answer for you?

The right answer depends on your circumstances. Many programmes allow retakes but have rules in place to limit the delivery rate of the assessment in order to limit content exposure.

1. You should communicate your retake policy to participants and to stakeholders who see the results of the assessments.

2. If you release scores, you also need to decide whether you will have a policy  as to whether scores for all attempts are released, or (as many organizations do) only for the successful attempt. Section 11.2 of the the Standards for Educational and Psychological Testing states

“Test users or the sponsoring agency should explain to test takers their opportunities, if any, to retake an examination; users should also indicate whether the earlier as well as later scores will be reported to those entitled to receive score reports.”

3. You should not allow people to retake a test they have passed.

4. You should consider requiring a period of time to elapse before someone retakes an exam if they fail. This allows time for them to update their learning. You can easily set this up when scheduling within Questionmark software, for example the dialog below gives a 7-day gap.

Limit days between retakes

5. Unless special circumstances apply, you will usually want to allow at least one retake and probably at least two retakes.

6. You may want to consider some intervention or stop procedure after a certain number of failed attempts. A common number I’ve heard anecdotally is three attempts, but it will depend on each assessment program’s own individual factors and use cases.  If this is an internal compliance exam, you might want to organize some remedial training or job review. If this is a public exam, you might want to ensure a longer time period to allow reflection and re-learning.

Please feel free to comment below if you have alternative thoughts on the number of retakes to allow.

To Your Health! Good practice for competency testing in laboratories

John Kleeman HeadshotPosted by John Kleeman

In the world of health care, from pathology labs to medical practitioners to pharmaceutical manufacturers, a mistake can mean much more than a regulatory fine or losing money – people’s lives and health are at stake. Hospitals, laboratories and other medical organizations have large numbers of people and need effective systems to make them work well together.

I’ve been learning about how assessments are used in the health care sector. Here is the first of a series of blog articles in the  theme of “learning from health care”.

In this article, I’d like to share some of what I’ve learned about how pathology and other health care laboratories approach competency assessment. Laboratory personnel have to work tirelessly and in an error-free way to give good quality, reliable pathology results. And mistakes cost – as the US College of American Pathologists (CAP) state in their trademarked motto “Every number is a life”. I think there is a lot we can all learn from how they do competency testing.

Job Description -> Task-specific Training -> Competency Assessment -> Competency RecognitionA good place to start is with the World Health Organization (WHO). Their training on personnel management reminds us that “personnel are the most important laboratory resource” and they promote competency assessment based on a job description and task-specific training as shown in the diagram on the right.

WHO advise that competency assessments should be conducted regularly (usually once or twice a year) and they recommend observational assessments for many areas of competence:  “Observation is the most time-consuming way to assess employee competence, but this method is advised when assessing the areas that may have a higher impact on patient care.” Their key steps for conducting observational assessments are:

  • Assessor arranges with employee a pre-arranged time for the assessment
  • The assessment is done on routine work tasks
  • To avoid subjectivity, the assessment should be recorded on a fixed check-list with everyone assessed the same way, to avoid bias
  • The results of the assessment are recorded, kept confidential but shared with the employee
  • If remediation is needed, an action plan involving retraining is defined and agreed with the employee

WHO’s guidance is international. Here is some additional guidance from the US, from a 2012 presentation in the US by CAP’s inspections team lead on competency assessment for pathology labs. This advice seems to make sense in a wider context:

  • If it’s not documented, it didn’t happen!
  • You need to do competency assessment on every person on every important system they work with
  • If employees who are not in your department or organization, contribute significantly to the work product, you  need to assess their competence too. Otherwise the quality of your work product is impacted
  • Competency assessment often contains quizzes/tests, observational assessments, review of records, demonstration of taking corrective action and troubleshooting
  • If people fail competency assessment, you need to re-train, re-assess and document that

If your organization relies on employees working accurately, I hope this provides value and interest to you. I will share more of what I’m learning in future articles.

How do you demonstrate to a regulator that your employees are competent?

Posted by John Kleeman

Many regulatory regimes round the world require companies to ensure that their employees are “competent”. This makes sense, as if you consider the alternative, no company wants to have incompetent employees! But there can be challenges as regulators aren’t always clear about what competence means, or how much training or assessment you need to do. And the consequences for making mistakes can be severe.

A new set of guidelines for good quality Training and Competence schemes has just been produced by the Financial Skills Partnership, a UK employer-led organization that works to enhance professionalism and talent in financial services. This is a 100-page document you can download for free with registration here.

If you work in Compliance or Training in UK financial services, this is a must-read. If you work in Compliance or Training  in financial services elsewhere, or if you’re in another industry where your regulator requires you to train employees to ensure competence, it’s worth a look.

The document works in a framework of regulator guidance to business strategy to operational practice, as shown below:From Financial Skills Partnership Guidance

There is general guidance and then more specific case studies.  For example, regarding how often you should assess competence, the guidance states rather generally:

How often competence is assessed should be determined according to the nature of the role and the business model it supports, and be included in the Training & Competence policy requirements. … A firm should take a risk based approach to decide the frequency of any tests or assessments that need to be undertaken.

But then one case study suggests competence tests when an employee joins and annually thereafter, and another case study suggests quarterly testing for all employees. So the guidelines are not prescriptive regarding what must be done but give useful guidance about what to ensure is in your policy and the range of things that are likely to be acceptable to a regulator.

(If you’re interested in the use of online assessments for compliance, you are welcome to download our white paper on the subject.)