Posted by John Kleeman
I’ve been learning how the railway industry uses assessments to maintain competence. As a frequent train passenger, it’s reassuring that the industry and their regulators carefully enforce a safety first mantra. And how railway and rapid transit companies (sometimes with Questionmark software!) use assessments to check competency for rail workers, especially in safety-critical roles.
There are many government and industry bodies that oversee and promote safety including the US Federal Railroad Administration, the UK Railway Safety & Standards Board (RSSB) and the European Railway Agency.
The RSSB have produced a very useful document: Good practice on Competence Review and Assessments. Here is a table summarized from this document that gives pros and cons of different kinds of assessment.
|Type of assessment||What it means||Pros||Cons|
|Observational||Observer watches participant doing normal work||Valid and reliable as it provides first-hand information about performance in real conditionsCaptures information about process and behavior not just outcomes||Risk of a ‘special performance’ as someone behaves differently whilst being observedWill not cover emergencies and other non-routine work
Needs good planning
|Simulation||Participant completes activity which is not real work but replicates real work closely||Provides performance evidence for non-routine workMeasures response to emergencies||Heavy on resourcesNeeds careful planning to be valid and reliable|
|Tests||Formal assessment of knowledge on paper or on screen||Consistent and objectiveGood for assessing technical knowledge
Cost effective for large numbers of people
|Requires skill to make valid and reliable|
|Work products||Examining outcomes of work done, e.g. document written or machine serviced||Provides evidence of performance in real work conditions||Need to verify authenticityShows outcome but not route to get there|
|Written reports||Report from participant or colleague describing competence on the job||Provides evidence to support other methods||Need to check authenticityMemory fallible
Requires writing skills
|Oral interview||Conversation where performance is described and questioned||Allows in depth exploration of knowledge and understanding||Relies on skills of assessorsHard to make consistent and objective|
It’s interesting to see the strengths and weaknesses of different kinds of assessments from such a safety-focused industry and to consider how using different assessments together can cover more ground and reduce safety risks.