Judgement is at the Heart of nearly every Business Scandal: How can we Assess it?

Posted by John Kleeman
How does an organization protect itself from serious mistakes and resultant corporate fines?

An excellent Ernst & Young report on risk reduction explains that an organization needs rules and that they are immensely important in defining the parameters in which teams and individuals operate. But the report suggests that rules alone are not enough, it’s how they are adopted by people when making decisions that matter. Culture is a key part of such decision making. And that ultimately when things go wrong “judgement is at the heart of nearly every business scandal that ever occurred”.

Clearly judgement is important for almost every job role and not just to prevent scandals but to improve results. But how do you measure it? Is it possible to test individuals to identify how they would react in dilemmas and what judgement that would apply? And is it possible to survey an organization to discover what people think their peers would do in difficult situations?  One answer to these questions is that you can use Situational Judgement Assessments (SJAs) to measure judgement, both for individuals and across an organization.

Questionmark has published a white paper on Situational Judgement Assessments, written by myself and Eugene Burke. The white paper describes how to assess judgement where you:

  1. Identify job roles and competencies or aspects of that role in your organization or workforce where judgement is important.
  2. Identify dilemmas which are relevant to your organization and each of which requires a choice to be made and where that choice is linked to the relevant job role.
  3. Build questions based on the dilemmas which asks someone to select from the choices –   SJA (Situational Judgement Assessment) questions.

There are two ways of presenting such questions, either to survey someone or to assess individuals on their judgement.

  • You can present the dilemma and survey your workforce on how they think others would do in such a situation. For example “Rate how you think people in the organization are likely to behave in a situation like this. Use the following scale to rate each of the options below: 1 = Very Unlikely 2 = Unlikely 3 = Neutral 4 = Likely 5 = Very Likely”.
  • You can present the dilemma and test individuals on what they personally would do in such a situation, for example as shown in the screenshot below.

You work in the back office in the team approving new customers, ensuring that the organization’s procedures have been followed (such as credit rating and know your customer). Your manager is away on holiday this week. A senior manager in the company (three levels above you) comes into your office and says that there is an important new customer who needs to be approved today. They want to place a big order, and he can vouch that the customer is good. You review the customer details, and one piece of information required by your procedures is not present. You tell the senior manager and he says not to worry, he is vouching for the customer. You know this senior manager by reputation and have heard that he got a colleague fired a few months ago when she didn’t do what he asked. You would: A. Take the senior manager’s word and approve the customer B. Call your manager’s cellphone and interrupt her holiday to get advice C. Tell the manager you cannot approve the customer without the information needed D. Ask the manager for signed written instructions to override standard procedures to allow you to approve the customer

You can see this question “live” with other examples of SJA questions in one of our example assessments on the Questionmark website at www.questionmark.com/go/example-sja.

Once you deliver such questions, you can easily report on the results segmented by attributes of participants (such as business function, location and seniority as well as demographics such as age, gender and tenure). Such reports can help indicate whether compliance will be acted out in the workplace, evaluate where compliance professionals need to focus their efforts and measure whether compliance programs are gaining traction.

SJAs can be extremely useful as a tool in a compliance programme to reduce regulatory risk. If you’re interesting in learning more about SJAs, read Questionmark’s white paper “Assessing for Situational Judgment”, available free (with registration) at https://www.questionmark.com/sja-whitepaper.

Top 10 uses of assessments for compliance

Headshot JuliePosted by Julie Delazyn

We recently announced a webinar on September 18th about why it’s good to use assessments for compliance.

Today I’d like to focus on how to use them, particularly within  financial services organizations – for whom mitigating risk of non-compliance is essential.

You can find out more about these in a complimentary white paper that highlights good practices in using assessments for regulatory compliance: The Role of Assessments in Mitigating Risk for Financial Services Organizations. But here, for quick reference, are ten of the most useful applications of assessment in a compliance program:

1) Internal exams — Internal competency exams are the most commonly used assessments in financial services.

2) Knowledge checks — It’s common to give knowledge checks or post-course tests (also called Level 2s) immediately after training to ensure that the training has been understood and to help reduce forgetting. These assessments confirm learning and document understanding.

3) Needs analysis / diagnostic tests — These tests measure employee’s current skills in topics and help drive decisions on development topics. They can be used to allow employees to test out when it’s clear they already understand a particular subject.

4) Observational assessments — When checking practical skills, it’s common to have an observer monitor employees to see if they are following correct procedures. A key advantage of an observational assessment is that it measures behavior, not just knowledge. Using mobile devices for these assessments streamlines the process.

5) Course evaluation surveys — “Level 1” or “smile sheet” surveys let you check employee reaction following training. They are a key step in evaluating training effectiveness. In the compliance field You can use them to gather qualitative information on topics, such as how well policies are applied in the field. Here’s an example fragment from a course evaluation survey.

eval survey6) Employee attitude surveys — Commonly used by HR for measuring employee satisfaction, these surveys also can be used to determine attitudes about ethical and cultural issues.

7) Job task analysis surveys –How do you know that your competency assessments are valid and that they are addressing what is really needed for competence in a job role? A job task analysis (JTA) survey asks people who are experts in a job how important the task is for the job role and how often it is done. Analysis of JTA data lets you weight the number of questions associated with topics and tasks so that a competency test fairly measures the importance of different elements of a job role.

8) Practice tests — Practice tests often use questions that are retired from the exam question pool but remain valid. Practice tests are usually accompanied by question and topic feedback. As well as allowing candidates to assess their further study needs, practice tests give candidates experience with the technology and user interface before they take a real exam.

9) Formative quizzes— These quizzes are those we are all familiar with: during learning to inform instructors and learners about whether learners have understood the learning or need deeper instruction, they diagnose misconceptions and also help reduce forgetting. They provide the key evidence that helps instructors vary the pace of learning. Computerized formative quizzes are especially useful in remote or e-learning where an instructor cannot interact face-to-face with learners.

10) 360- degree assessments – This kind of assessment solicits opinions about an employee’s competencies from his/her superiors, reports and peers. It will usually cover job-specific competencies and general competencies such as integrity and communication skills. In compliance, such surveys allow you to potentially identify issues in people’s behavior and competencies that need review.

Click here for details and registration for the webinar, 7 Reasons to Use Online Assessments for Compliance.

You can download the white paper, The Role of Assessments in Mitigating Risk for Financial Services Organizations, here.

“It takes 20 years to build a reputation and five minutes to ruin it”

John Kleeman HeadshotPosted by John Kleeman

A recent corporate survey reported in Insurance Journal suggests that reputation is the hardest risk to manage. The survey indicates that 81% of companies see reputation as their most significant asset but are challenged in knowing how to protect it.

Warren Buffett famously said, “It takes 20 years to build a reputation and five minutes to ruin it”. So how can organization’s avoid those fateful five minutes?

Assessments can be a great tool to mitigate risk to reputation. I’d like to share some ideas on this from my Questionmark colleagues, Eric Shepherd and Brian McNamara.

Let’s start by considering the classic business model shown in the diagram below. A company uses its core capabilities in Production with a supplier network and product/services development to make an Offer to its customers, which it communicates via a sales and marketing Channel, with a supporting Finance structure.

Classic business model. Production, Offer, Channel, Finance

The pink shaded areas in the diagram are where there is reputation risk.

If you make mistakes within Production – in regulatory compliance, processes & procedures or health & safety – this can seriously hurt your reputation. Errors in regulatory compliance or failing to follow processes & procedures can similarly damage reputation in Finance. Assessments can help by confirming health and safety, checking the competence of employees and testing knowledge of processes & procedures.

Many companies have a bigger challenge in the sales and marketing Channel, as this is more spread out and harder to control. You have to comply with laws on how you sell, both industry-specific and general ones like anti-corruption. The people in your Channel must have product/solution knowledge. And reputation is hurt by overselling and unsatisfied customers.

The diagram below breaks down the Channel into typical parts:

breaking down Channel into Market Messaging and Relationship Management

How can assessments help with reputation challenges in the Channel?

Market Messaging

When you message your customers, there is a risk that your messaging is inappropriate or that messages do not resonate. Most organizations assess customers with surveys to determine if they are “getting it”.


You need your sales people, whether in-house personnel or partners, to comply with laws and avoid corruption. They need to ensure your customers are satisfied, by selling fairly and not using trickery. Online quizzes and tests are great ways to check your sales people know all the rules and are competent to sell. Observational assessments using tablets or smartphones also let supervisors check team members.

Customer Care

In customer care, a challenge is high staff turnover, requiring lots of training. As in sales, the customer care team need product and process knowledge and need to satisfy customers. Quizzes and tests motivate learning, maintain focus and enable recognition of people who “get it”.

Technical Support

Lastly, every company has challenges when products or services don’t work. How you deal with problems impacts your reputation. The challenge for technical support is to delight the customer and fix problems on first call.

Quizzes and tests are useful in technical support, but something that works really well for technical teams is a certification program. Skills and knowledge required are often complex, and using assessments to certify gives technical support teams career progression. It also encourages pride in their jobs, leading to better employee retention and better service.


I hope this article helps you realize that online assessments help solve one of the biggest challenges facing business – mitigating risk to reputation. Next time you are making an internal case for online assessments, consider whether your senior management might find reducing reputation risk a compelling reason to deploy assessments.

Tips for delivering effective compliance assessments

Headshot Julie

Posted by Julie Delazyn

Our white paper, The Role of Assessments in Mitigating Risk for Financial Services Organizations, describes five stages of deploying legally defensible assessments:

Compliance five steps

I’ve shared pointers about Planning, Deployment and Authoring in previous posts. For today, here are some tips about Delivery.

Some of these recommendations are specific to Questionmark technologies, but most can be applied to any testing and assessment system:

Delivery chartClick here to read the paper, which you can download free after login or sign-up.

If you are interested in this topic and are eager to learn more about the effective use of online assessments, join us at the Questionmark Users Conference for a full program of cases studies, discussions, presentations about best practices and instruction in the use of the latest Questionmark technologies. Sign up while there’s still time. See you in Baltimore!


Planning compliance-related assessments: good practice recommendations

Headshot JuliePosted by Julie Delazyn

Last week I wrote about the business benefits of assessments that mitigate risk and help ensure compliance– as described in our white paper, The Role of Assessments in Mitigating Risk for Financial Services Organizations, This paper offers a great deal of information about these kinds of assessments and advice about best practices for implementing a legally defensible assessment program.

The paper describes five stages of deploying assessments — from planning to analytics — and offers recommendations for good practice for people in these six job roles:

  • Chief Compliance Officer, responsible for compliance in the organization
  • Compliance Officer, who runs the assessment part of the compliance program
  • Authoring Expert, the authoring team lead
  • SME, the Subject matter expert who authors and reviews questions
  • Trainer, who trains in compliance
  • IT Specialist, responsible for IT setup

Compliance five steps

Some of these recommendations are specific to Questionmark technologies, but most can be applied to any testing and assessment system.

Today, let’s look at good practice for the first of the five stages: planning:

Planning - Compliance

Click here to read the paper, which you can download free after login or sign-up.

Testing out of training: It can save time and money

Posted by Julie Delazyn

If people already know something well, then teaching them about it is a waste of resources and motivation. That’s where “testing out” comes in. Diagnostic tests can provide a way of identifying what employees know and determining if they need further training.

Offering training needs assessments enables employees who can demonstrate that they already understand a subject to “test out” of unnecessary training on it. This saves time for both training departments and employees. And, of course, that translates into cost savings, too. At the Questionmark 2011 Users Conference, for instance, a large accountancy firm reported saving more than 500 hours of training thanks to a diagnostic test, with savings of more than US$40,000.

Is testing out a supported practice? A Training Room article by Meg Sczyrba in the American Banking Association Bank Compliance magazine reported a consensus among several US regulators about the acceptability of testing out, provided the testing program is well-structured.

To read more about effective ways to use diagnostic and other types of assessments in the context of compliance, check out our white paper: The Role of Assessments in Mitigating Risk for Financial Services Organizations. You can download it free here, after login.