Where do you get evidence for training effectiveness? Ideas from the A-model

Posted by John Kleeman

How do you design a training intervention to improve performance? And how do you gather effective evidence to show that the intervention is effective?

In a couple of previous articles, Problem, Performance and Program in the A-model and A whistle-stop tour round the A-model, I’ve described how the A-model helps you do this by separating Problem, Performance and Program; and then by working through these to analyze and design the program and then assess and evaluate it.

Whether you are using the A-model, or one of the other models of showing training effectiveness (Kirkpatrick, Phillips, Bersin etc.), you need good data to be able to show effectiveness. How and where do you get this data?

A neat concept in the A-model is the evaluation cube shown below, which identifies 3x3x3 different ways in which you can collect data to feed into the A-model.

The evaluation cube

The evaluation cube shows 3 dimensions of possible evaluation

You can evaluate:

  • Problem – the business problem that you’re seeking to address (e.g. increase sales or improve productivity)
  • Performance – people’s performance in job roles (e.g. how someone’s capability changes or improves)
  • Program – the training or learning intervention and how successful it is (how a specific intervention acts)

With data from:

  • The individual (e.g. tests or surveys taken by people to measure knowledge, skills, performance or attitudes)
  • Observers (assessments where someone rates another, for example an observational or workplace assessment of skills, attitude or performance)
  • Systems (gathering data from records. e.g. sales or accounting or HR records or other systems

And you can collect data relative to:

  • Self – how someone’s performance e.g. on a test changes over time, for instance from pre-training to post-training or from one year to another
  • Others or the norm – how data about a person or group compares to the distribution of others, for example how this department or person compares to other departments or people, or the norm for the organization
  • Criteria – how data compares against targets or benchmarks, absolute measures

For example, if you are looking to identify data about Performance, you could consider 9 facets of the cube: individual/observers/systems vs self/others/criteria. And you might use an observational assessment before and after the learning intervention (observers/self) to identify change and also a test of an aspect of performance after the event (individual/others) to measure capability compared to the norm for the organization.

Thinking about the different facets of the cube can be a good way to brainstorm ideas for assessments and data gathering.

I hope these ideas spur you to effective measurement of training interventions, whatever methodology you are using.

For more on the A-model, see Dr. Bruce C. Aaron’s excellent white paper, available here (free with registration).

Early-bird alert: Last day for saving on conference registration

Posted by Joan Phaup

This is the last time I’ll mention early-bird sign-ups for the Questionmark 2012 Users Conference.

That’s because today is the last day to save $100 on your conference registration.

We are looking forward to seeing customers in New Orleans March 20 – 23 for three days of exceptional learning, professional development and networking.

A few highlights from the conference program:

Why wait any longer? Sign up today!

Blackboard Connector – Instructor Perspective

Posted by Steve Lay

In this video, I provide an overview of the Blackboard Connector, as experienced by a typical course instructor. I’ll show you how to create content items that enable you to seamlessly embed Questionmark assessments and surveys into your Blackboard courses. I’ll also explain the way user synchronisation works, and how to use the course control panel to view coaching reports and log in to Enterprise Manager.
In my next video I’ll provide an overview of the connector from the point of view of the Blackboard System Administrator.

Questionmark Live Now Includes the Numeric Question Type

Posted By Doug Peterson

Questionmark Live continues to grow and add new functionality!

In this video, I will take you on a tour of the latest addition to Questionmark Live, the Numeric question type. Not only can you create a simple question with a single numeric answer, you can create questions that require multiple numeric answers — something that’s a bit complicated to do in Authoring Manager but very easy to do in Questionmark Live!

Conference Close-up: Questionmark Boot Camp for Beginners

Posted by Joan Phaup

For people attending the Questionmark Users Conference who are new to our software, we’re pleased to be offering a one-day pre-conference workshop that will get them off to a great start!

Our own Rick Ault, who has led many a Questionmark product training course, will lead Questionmark Boot Camp: Basic Training for Beginners at the Ritz Carlton New Orleans on Tuesday, March 20. He shared some details about this with me a few days ago:

“Boot Camp” sounds a little scary! What will happen during that workshop?

We call it boot camp in the same sense that a military boot camp gives you basic training in order to get more specialized skills. Our boot camp will give new Questionmark users the basic skills and knowledge so that they can get as much as possible out of the conference. We’ll give people a  solid foundation, so that they can seek more specialized information and drill a little deeper during the conference into the areas that interest them most. We want to empower attendees to make the most of their conference.

Rick Ault

Rick Ault

How much do you expect people  to learn in one day?

They’ll get a broad overview of how things work and pick up some skills by actually doing things and getting some practice. We’ll cover basic concepts such as how you create a question, how you group questions into an assessments, then publish it, schedule it, deliver it and get the results back. From start to finish, they’ll bulid up a foundation that will help them have meaningful discussions during the conference. We’re distilling our regular three-day training course into one day, so it’s not nearly as deep, but the scope of the class is the same as the longer course.

What do you hope people will take away from their day in class?

I hope they’ll take away a good understanding of how Questionmark works. I hope they’ll be able to go back to work being able to create some basic assessments, and that they’ll be on their way to doing more advanced things if they decide to go that route.

How should participants prepare for Boot Camp?

Get a good night’s rest, bring your computer and make sure it works!

What are you looking forward to at the conference?

It’s always nice to shepherd my flock – to see my past training attendees and see how they’ve grown and how they’re using Perception today…They teach me new things sometimes!

If you have not signed up for the conference yet, we hope you will sign up in the coming week to save $100 on your registration fee! (Early-bird registration ends next Friday, January 27.)

Top 5 Questionmark Presentations on SlideShare

Posted by Julie Delazyn

Presentations make up a huge part of the way thought leaders at Questionmark pass on all the great information that comes from our various white papers, research and case studies.

We have been sharing many of these presentations with you by featuring and posting them to our new Questionmark SlideShare page.  I read and answer comments all the time from LinkedIn, Facebook, Twitter and, most recently, Google+ about the value of our presentations as well as the ways in which they are being shared and used. These are in high demand, and last week SlideShare featured Questionmark on its homepage for having the most popular presentation on that particular day.

So, we thought it would be helpful to highlight our five most popular presentations on our SlideShare page (in no particular order):

Feel free to comment, share and let us know in which ways these have helped you!