Ten top tips for effective online assessment from HSBC’s Matt Bushby

Posted by John Kleeman

HSBC – a multinational banking and financial services organisation – uses Questionmark Perception worldwide for delivering computer-based, online competency tests for employees to check they understand products and regulations. Matt Bushby is an Assessment Specialist and the Global Subject Matter Expert for Questionmark Perception at HSBC.

Matt has been responsible for the transfer of paper-based testing to online assessment in the UK since 2006 and has supported and advised a dozen or so other HSBC country teams in their move from different tools and systems to a joined-up, single assessment system from Questionmark. By using assessments in this way, HSBC are able to meet strict financial regulations in each country and also ensure a premium service to customers.

Matt Bushby and his lovely children!

I asked him to share from his experience some key tips for making online assessment effective within a large company, and here is his advice:

1. Start with “why” you are assessing. You need to have an informed conversation within the business to get discussion and buy-in. Some typical questions which can stimulate the conversation:

· How many people are likely to take the assessment?

· What are the consequences for failing?

· How high or low stakes is the assessment?

· Can people re-take the test?

· What follow-on learning will you use for people who pass the test but get some questions wrong?

2. Work out some key policy questions for the assessment:

· Should it be open-book or closed-book?

· Should everyone see the same questions or should there be random selection?

· Should there be a time limit or should it be untimed?

· Do participants have to answer all questions or can they leave some blank?

· Should the assessment give feedback on wrong answers?

3. Plan the assessment before you start writing the questions. Work out the topic granularity and structure and how many questions you need in each topic. For example, a typical assessment will call in questions from many different topics and for many assessments. HSBC typically create 3 spare questions for each question actually delivered to a test-taker.

4. Agree with the business how the structure of the assessment is going to be validated and agreed, both initially and on-going as the business shifts focus over time. Typically this would be by the business reviewing the overview blueprint for an assessment.

5. HSBC finds that multiple choice questions work well for much competency assessment, providing they are written well. It’s key to provide guidance to subject matter experts in question design. Here are some key points :

· have a consistent number of choices,

· make all incorrect choices reasonable,

· avoid negative language,

· don’t make the correct choice the longest!

6. When it meets the business need, encourage scenario questions that test application rather than factual questions. Scenarios can work well as multiple choice questions, where distractors describe common mistakes. You may need to allow longer time limits for assessments with scenario questions.

7. Be careful about using non-accessible visual elements like screenshot images in questions, as any large company will likely have visually impaired employees or others with accessibility needs that need to be catered for. It’s usually best to have one assessment that all can take, rather than having to make variants.

8. Matt finds giving generic feedback to read up on a topic or syllabus area very effective, as this means that he doesn’t need to include the original question in the feedback. This also means less risk of leakage via sharing feedback between participants.

9. For assessments that are always live (for example assessments as part of on-boarding training), keep in touch with subject matter experts and the business to ensure that these are regularly reviewed and stay up to date.

10. When introducing a new assessment, follow the approach of pilot / review / amend / launch.

Conference Close-up: A corporate perspective on global learning

Posted by Jane Townsend

As the Questionmark European Users Conference draws closer, we are looking forward to many customer presentations along with sessions on using Questionmark technologies effectively and following best practices.

We’re pleased to have Matt Bushby from HSBC – a multinational banking and financial services organisation —  on the conference program. Matt will present a case study on the challenges HSBC faced consolidating its global learning programs while at the same time upgrading to a newer version of Questionmark Perception. This new system will put content at the fingertips of employees throughout the world, but a secure, integrated deployment on such a large scale presents some special challenges. I talked with Matt this week to find out about his plans for the conference.

Matt Bushby and friends!

Tell me about your work at HSBC.

I am the UK assessment specialist, which means I’m responsible for the consultancy, design and build of online assessments within HSBC’s UK bank. Any assessment we do, any or question we do, basically comes across my desk. I’m also the global subject matter expert for Questionmark within HSBC, so if anyone has any issues, has a problem or wants to know how to do something, they come to me.

So you’re very, very busy!

Yes, that’s the word I’d use!

How are you using assessments at the bank?

Within the UK we use it for Regulatory testing. So job roles that fall under the governance of the Financial Services Authority take compliance assessments via Questionmark, through our SumTotal learning management system. We also use Questionmark assessments for non-regulatory, high-stakes testing too.  Elsewhere in the HSBC Group, some of our people also use Questionmark for quizzes, but here in the UK we focus on higher-stakes, secure assessments.

What are you going to talk about at the conference?

I’ll be describing the special challenges of working with a corporate environment – from the point of view of a large multinational corporation. In addition to upgrading to a new version of Questionmark Perception, we joined up numerous different learning management systems into one global system. This has will enable the sharing of content in a far more effective way, but it was a massive task.

The challenges involve how you scale the system so people still have access to it while, at the same time, IT maintains centralised control of the servers. How do you efficiently update the database from one version of Perception to another? How do you manage multilingual assessments?  How do you handle test security? How can the global SME best share knowledge about assessment management  with colleagues in Hong Kong or South America? And what issues are involved in collaborating with test authors from many different countries?

I’ll talk about some of the pitfalls and challenges we’ve faced and the solutions we’ve worked out.  I’ll also talk about the support Questionmark has given us to make a template customisable for different languages.

What are you looking forward to about the conference?

The thing I like most about the conference is the networking. It’s sitting with other people. It’s talking about our experiences. It’s a chance to catch up with people from Questionmark. I have a really good working relationship with Questionmark support, and we talk on the phone – but it’s important to take time to put faces to names, to link up, to work on the relationship. I like being involved with the product, the company and the customers – and to find out how they do things.

Click here to learn more about the conference and register soon!