Judgement is at the Heart of nearly every Business Scandal: How can we Assess it?

Posted by John Kleeman
How does an organization protect itself from serious mistakes and resultant corporate fines?

An excellent Ernst & Young report on risk reduction explains that an organization needs rules and that they are immensely important in defining the parameters in which teams and individuals operate. But the report suggests that rules alone are not enough, it’s how they are adopted by people when making decisions that matter. Culture is a key part of such decision making. And that ultimately when things go wrong “judgement is at the heart of nearly every business scandal that ever occurred”.

Clearly judgement is important for almost every job role and not just to prevent scandals but to improve results. But how do you measure it? Is it possible to test individuals to identify how they would react in dilemmas and what judgement that would apply? And is it possible to survey an organization to discover what people think their peers would do in difficult situations?  One answer to these questions is that you can use Situational Judgement Assessments (SJAs) to measure judgement, both for individuals and across an organization.

Questionmark has published a white paper on Situational Judgement Assessments, written by myself and Eugene Burke. The white paper describes how to assess judgement where you:

  1. Identify job roles and competencies or aspects of that role in your organization or workforce where judgement is important.
  2. Identify dilemmas which are relevant to your organization and each of which requires a choice to be made and where that choice is linked to the relevant job role.
  3. Build questions based on the dilemmas which asks someone to select from the choices –   SJA (Situational Judgement Assessment) questions.

There are two ways of presenting such questions, either to survey someone or to assess individuals on their judgement.

  • You can present the dilemma and survey your workforce on how they think others would do in such a situation. For example “Rate how you think people in the organization are likely to behave in a situation like this. Use the following scale to rate each of the options below: 1 = Very Unlikely 2 = Unlikely 3 = Neutral 4 = Likely 5 = Very Likely”.
  • You can present the dilemma and test individuals on what they personally would do in such a situation, for example as shown in the screenshot below.

You work in the back office in the team approving new customers, ensuring that the organization’s procedures have been followed (such as credit rating and know your customer). Your manager is away on holiday this week. A senior manager in the company (three levels above you) comes into your office and says that there is an important new customer who needs to be approved today. They want to place a big order, and he can vouch that the customer is good. You review the customer details, and one piece of information required by your procedures is not present. You tell the senior manager and he says not to worry, he is vouching for the customer. You know this senior manager by reputation and have heard that he got a colleague fired a few months ago when she didn’t do what he asked. You would: A. Take the senior manager’s word and approve the customer B. Call your manager’s cellphone and interrupt her holiday to get advice C. Tell the manager you cannot approve the customer without the information needed D. Ask the manager for signed written instructions to override standard procedures to allow you to approve the customer

You can see this question “live” with other examples of SJA questions in one of our example assessments on the Questionmark website at www.questionmark.com/go/example-sja.

Once you deliver such questions, you can easily report on the results segmented by attributes of participants (such as business function, location and seniority as well as demographics such as age, gender and tenure). Such reports can help indicate whether compliance will be acted out in the workplace, evaluate where compliance professionals need to focus their efforts and measure whether compliance programs are gaining traction.

SJAs can be extremely useful as a tool in a compliance programme to reduce regulatory risk. If you’re interesting in learning more about SJAs, read Questionmark’s white paper “Assessing for Situational Judgment”, available free (with registration) at https://www.questionmark.com/sja-whitepaper.

New White Paper Examines how to Assess for Situational Judgment

Posted by John Kleeman

Is exercising judgment a critical factor in the competence of the employees and contractors who service your organization? If the answer to this is yes, as it most likely is, you may be interested in Questionmark’s white paper, just published this week on “Assessing for Situational Judgment”.

It’s not just CEOs who need to exercise judgment and make decisions, almost every job requires an element of judgment. Situational Judgment Assessments (SJAs) present a dilemma to the participant and ask them to choose options in response.


Context is defined -> There is a dilemma that needs judgment -> The participant chooses from options -> A score or evaluation is made

Here is an example: 

You work as part of a technical support team that produces work internally for an organization. You have noticed that often work is not performed correctly or a step has been omitted from a procedure. You are aware that some individuals are more at fault than others as they do not make the effort to produce high quality results and they work in a disorganized way. What do you see as the most effective and the least effective responses to this situation?
A.  Explain to your team why these procedures are important and what the consequences are of not performing these correctly.
B.  Try to arrange for your team to observe another team in the organisation who produce high quality work.
C.  Check your own work and that of everyone else in the team to make sure any errors are found.
D.  Suggest that the team tries many different ways to approach their work to see if they can find a method where fewer mistakes are made.

In this example, option C deals with errors but is time consuming and doesn’t address the behavior of team members. Option B is also reasonable but doesn’t deal with the issue immediately and may not address the team’s disorganized approach. Option D is asking a disorganized team to engage in a set of experiments that could increase rather than reduce errors in the work produced. This is likely to be the least effective of the options presented. Option A does require some confidence in dealing with potential pushback from the other team members, but is most likely to have a positive effect.

You can see some more SJA examples at http://www.questionmark.com/go/example-sja.

SJA items assess judgment and variations can be used in pre-hire, post-hire training, for compliance and for certification. SJAs offer assessment programs the opportunity to move beyond assessments of what people know (knowledge of what) to assessments of how that knowledge will be applied in the workplace (knowledge of how).

Questionmark’s white paper is written as a collaboration by Eugene Burke, well known advisor on talent, assessment and analytics and myself. The white paper is aimed at:

  • Psychometricians, testing professionals, work psychologists and consultants who currently create SJAs for workplace use (pre-hire or post-hire) and want to consider using Questionmark technology for such use
  • Trainers, recruiters and compliance managers in corporations and government looking to use SJAs to evaluate personnel
  • High-tech or similar certification organizations looking to add SJAs to increase the performance realism and validity of their exam

The 40 page white paper includes sections on:

  • Why consider assessing for situational judgment
  • What is an SJA?
  • Pre-hire and helping employers and job applicants make better decisions
  • Post-hire and using SJAs in workforce training and development
  • SJAs in certification programs
  • SJAs in support of compliance programs
  • Constructing SJAs
  • Pitfalls to avoid
  • Leveraging technology to maximize the value of SJAs

Situational Judgment Assessments are an effective means of measuring judgment and the white paper provides a rationale and blueprint to make it happen. The white paper is available free (with registration) from https://www.questionmark.com/sja-whitepaper.

I will also be presenting a session about SJAs in March at the Questionmark Conference 2018 in Savannah, Georgia – visit the conference website for more details.

Badging and Assessment: If they know it, let them show it!

Posted by Brian McNamara

We are delighted to announce the availability of Questionmark Badging!

With Questionmark Badging and Questionmark OnDemand, you can grant “badges” to participants based on the outcomes achieved on assessments such as certification exams, post-course tests or advancement exams.Badges associated with Questionmark assessments provide participants with portable, verifiable digital credentials.

Badges aligned with Questionmark assessments can be tied in with competencies and achievements, helping organizations provide recognition and motivation for increasing knowledge and skills. For credentialing and awarding bodies, they can increase the visibility and value of certification programs.

The new app couples Questionmark’s capabilities in delivering valid, reliable and trustworthy assessments with the industry-leading digital credentialing platform from Credly. More than just a visual representation of accomplishment, digital badges provide participants with verifiable, portable credentials that can be shared and displayed across the web, including social networking sites such as LinkedIn, Facebook and Twitter.

Find more info about Questionmark Badging right here!

Twelve tips to make questions translation ready

Posted by John Kleeman

We all know the perils of mis-translation. My favourite mis-translation is the perhaps apocryphal tale of a laundry in Rome, Italy putting up a sign in English saying “Ladies, leave your clothes here and spend the afternoon having a good time.” With Questionmark having a translation management system to help you translate questions and assessments, here are some good practice tips on writing questions so they will be easy to translate.

1. Avoid questions that assume syntax is the same in all languages, for example fill-in-blank questions that rely on word order. For example, in English, the verb goes in the middle of a sentence but in Turkish and Korean, the verb is usually at the end of a sentence.

2. Also avoid “broken stem questions”, where the stem is an incomplete sentence and the participant must select the most appropriate answer to finish the sentence. That’s likely to be challenging to translate in some languages where the ordering may not make sense.

3. Keep questions simple. Avoid unnecessarily complex text or question stems with redundancy or unnecessary repetition. Such questions are best simplified before you translate them.

4. Avoid metaphors and idiomatic language in general; things like “in small steps” or “disappear into thin air”, could well introduce translation mistakes.

5. Avoid passive voice where you can. Not all languages make it easy to translate this, and it’s usually best to just use active voice.

6. Thoroughly review questions prior to translation to ensure no ambiguity. if the question wording is ambiguous, the translator’s interpretation of the question may not be the same as that of the question author.

7. If you are using a rating scale across many questions, investigate its cultural appropriateness and, if possible, whether it is widely used in the target language.

8. Test items based on nuances of vocabulary, descriptions of emotions or abstract concepts can be hard to translate, as different languages may have different vocabulary connotations.

9. You also need to be aware of the risk that translating a question could cause a question to become obvious due to different words in the target language, like the following Swedish example.

Question that shows a Swedish translation giving the answer to a question by words being the same

10. Avoid using cultural context within question stimulus. If you are presenting a scenario, make it one that is relevant to different cultures and languages. If it is difficult to avoid a culturally marked context, consider preparing good guidelines for translators in which you define what adaptations are encouraged, desirable and ruled out

11. If your question contains a graphic or video, consider if you can remove any text from it and still keep the question meaningful. Otherwise you need to translate the text in the graphic or video in each language.

12. if you are translating items into several languages, it can be cost effective to conduct a translatability assessment on the items before you do the detailed translation. This will alert to possible issues within various language families prior to doing the more substantial work of full translation. A translatability assessment lets you identify and fix issues early and relatively cheaply. See here for a blog from Steve Dept of CApStAn that explains more.

Thanks to Steve Dept for inspiring this blog post with an excellent conference presentation at EATP last year and for helping me write this article. For some more advice on translating and adapting tests, see the International Test Commission Guidelines for Translating and Adapting Tests or the Cross-cultural Survey Guidelines (CCSG), both of which have been recently updated.

I hope this advice helps you be efficient in your translation efforts. For information on Questionmark OnDemand which includes translation management system capabilities, see www.questionmark.com.

6 Tips for trustworthy compliance assessments

Posted by Chloe Mendonca

If you’re responsible for the development or management of compliance tests you have a heavy responsibility on your shoulders. It’s up to you to ensure your tests are both valid and reliable. We’ve spoken about reliability and validity many times here on the Questionmark blog and these really are two of the keys to ensuring your assessment results can be trusted. If your tests don’t measure what they’re designed to or the content doesn’t reflect the required job knowledge, how can you make defensible decisions on the basis of the results?

This infographic shares 6 tips that you should consider implementing if you haven’t already that will help you to develop trustworthy compliance assessments.

 

Click here to get a hi-res copy of this infographic.

To learn more about developing trustworthy assessments, check out the 26-page Questionmark White Paper “Assessment Results You Can Trust”.

The Ultimate Guide To Using Assessments for Compliance [eBook]

ebookJulie ProfilePosted by Julie Delazyn

With increasing regulatory requirements, compliance is becoming more and more of a priority for many organizations.

Without regular testing, how do you know what your employees know? And in the case of an audit or an emergency, is it good enough to have had the participant sign off saying that they’ve attended training and understand the content? Most organizations today see online assessments as a critical part of their compliance programs.

Download your complimentary copy of the eBook: Using Assessments for Regulatory Compliance to learn about the most useful applications of assessments in a compliance program and best practice recommendations for using them.