Using the Demographic Report

The Demographic Report was recently added to Questionmark Analytics. Here are the basics:

What it does: The demographic report breaks down results by demographic variables such as language,  course name, location, department, instructor, and job role or military rank. This report can recognize up to 10 demographic variables recorded by an  assessment. Users can review the performance of assessment data broken out by a demographic variable, using data stored in special fields within the database.

Who should use it: Assessment, learning and education professionals can use this report to zero in on test results according to specific demographic information.

How it looks: Graph 1 in this example shows the assessment mean score for the selected demographic. It displays the average percentage score achieved as well as the high and low results.  Graph 2 shows the number of results for the selected demographic and includes an overall calculation of the number of results found.

You can assign several filters to limit the information included in the report:

  • Assessment (mandatory)
  • Group
  • Date
  • Attempt
  • Special field

A whistle-stop tour round the A-model

Posted by John Kleeman

In an earlier blog, I described how the A-model starts with Problem, Performance and Program. But what is the  A-model? Here’s an overview.

It’s easy to explain why it’s called the A-model. The model (shown on the left) traces the progress from Analysis & Design to Assessment & Evaluation. When following the A-model, you move from the lower left corner of the “A” up to the delivery of the Program (at the top or apex of the “A”) and then down the right side of the model to evaluate how the Problem has been resolved.

The key logic in the model is that you work out the requirements for success in Analysis & Design, and then you assess against them in the Assessment & Evaluation phase.A-model overview

Analysis and Design

During Analysis & Design, you define the measures of success, including:

  • How can you know or measure if the problem is solved?
  • What performance are you looking for and how do you measure if it is achieved?
  • What are the requirements for the Program and how do you measure if they are met?

It’s crucial to be able to do this in order to do Assessment & Evaluation against what you’ve worked out in Analysis & Design. Assessments are useful in Analysis & Design – for example needs assessments, performance analysis surveys, job task analysis surveys and employee/customer opinion surveys.

Assessment & Evaluation of Program

A common way to evaluate the program is to administer surveys covering perceptions of satisfaction, relevance and intent to apply the Program in the workplace. This is like a “course evaluation survey” but it focuses on all the requirements for the Program. Evaluation of program delivery therefore also includes other factors identified in Analysis & Design that indicate whether the solution is delivered as planned (for example, whether the program is delivered to the intended target audience at the right time).

Assessment & Evaluation of Performance

The A-model suggests that in order to improve Performance, you identify Performance Enablement measures – enablers that are necessary to support the performance, typically learning, a skill, a new attitude, performance support or an incentive.

You may be able to measure the performance itself using business metrics like number of transactions processed or other productivity measures. Assessments can be useful to measure performance and performance enablers, for instance:Tests to assess knowledge and skill. For instance:

  • Tests to assess knowledge and skill
  • Observational assessments (e.g. a workplace supervisor assessing performance against a checklist)
  • 360 degree surveys of performance from peers, colleagues and managers

Measuring whether the problem is solved

How you measure whether the problem is solved will arise from the analysis and design done originally. A useful mechanism can be an impact survey or follow-up survey, but there should also be concrete business data to provide evidence that the problem has been solved or business performance improved.

Putting it all together: the A-model

The key in the A-model is to put it together, as shown in the diagram below. You define the requirements for the Problem, the Performance, Performance Enablement and the Program. Then you assess the outcomes – for the delivery of the program, for the Performance Enablement and the Performance itself and then for the Impact against the business.

A-model

I hope you enjoyed this whistle-stop tour. For a more thorough explanation of the A-model, read Dr. Bruce C. Aaron’s excellent white paper, available here (free with registration).

Hmmm…What’s in this parcel?

‘This the season, so have some fun with this little gift from Questionmark! Happy Holidays!

Scan the QR code or click here to get started!

Conference Close-up: Using the Net Promoter® Score in Training Evaluations

Joan PhaupPosted by Joan Phaup

Last week I mentioned a peer discussion at the upcoming Questionmark 2012 Users Conference about the relative merits of proctored and non-proctored assessments.

Today I’d like to share a conversation I had recently with Frank Loforte, who will be one of our case study presenters in New Orleans March 20 – 23. Frank works for Beckman Coulter, Inc., where he is involved in training and certifying the company’s technicians and customers in the use of the company’s biomedical testing products, During his presentation, he’ll be describing how his team is using the Net Promoter® Score in training evaluation surveys.

Tell me about your work at Beckman Coulter.

Frank Loforte

Frank Loforte

I do a little bit of everything, but my main job is principal technical trainer. I have been teaching service engineers how to use the instruments our company sells. Along with that I maintain the computers and the websites as well as running tests and surveys. That’s where Questionmark comes in: we use it primarily for our (Kirkpatrick Model) Level 1 and Level 2 assessments.

How are you using Questionmark?

We use it to monitor how the customers and service engineers who come through our training center respond to our courses. We track those surveys and evaluations to make sure they are pleased with the training. We also gauge our students’ levels of knowledge, application and analysis skills. We give a knowledge test, an application test and an analysis test, which we grade separately. Then we report those scores to their managers in the field. We also go back three months after a course to ask managers how well their students are performing, in order to get the supervisor’s point of view as well as the student’s point of view.

You’re going to be talking the Net Promoter Score at the Users Conference. Can you explain a little about that?

We have been using Net Promoter Score questions for about a year and half. This helps us track and quantify how many students are pleased with the training. Many companies use this kind of question for collecting people’s opinions about products. So when they sell a widget they ask customers if they would recommend the widget to someone else on a zero to 10 scale. People who respond with a score of 9 or a 10 are considered promoters. The 8’s and 7’s are called “passives” and those who rate something from 0 to 6 are called “detractors.”

The scoring is kind of complicated (and I’ll be explaining it during my session) but it gives you a really clear indication of how you are doing. Even more valuable is the follow-up question: “What is the most important reason for that score?” We look closely at the responses we get to that question from the detractors and respond to them right away.

What do you expect people to learn from your session?

I’d like them to be able to see what Net Promoter Score is, how we apply it how they can use it in their environments. We have a track record with it now, so I have a lot of data and it’s pretty consistent. It seems we are getting the same type of response from a varied public, so it’s a good indicator. It’s what you do with the answer to the second question that’s really going to be important: fixing the things they say are broken and continuing to do the things they like.

What are you looking forward to at the conference?

I want to learn about other people and what they do. That’s what’s really great: seeing people from all different companies using the same product and asking them questions. I also want to learn more about Questionmark Analytics and any new things that are coming along.

If you would like to learn more about the conference and register online, click here.

Improving Multiple-Choice Questions

Posted By Doug Peterson

I’ve heard a lot of criticisms of multiple-choice questions (MCQs) over the years.

For instance:

  • They really only test recall, not true understanding
  • They test reading comprehension as much, if not more so, than they test actual knowledge
  • They’re unfair to people with dyslexia
  • They imprint incorrect answers in the learner’s brain, which may be recalled in error at some point in the future
  • It’s easy to infer the correct answer from the recognition of keywords or the length of answers

But many of these problems might not be so much because of MCQs by definition: they may be due to  poorly written MCQs!

Let’s face it – we all know that MCQs are pretty much the most-used question type on the planet. Why?

  • They’re easy to use – they work online, on paper, etc.
  • They’re easy to score
    • Completely objective – anyone can do it
    • Easy to automate with bubble sheets and scanners

Given the popularity of multiple-choice questions, we can always do with good advice about how to improve them!

The latest issue of Learning Solutions Magazine has a very well-written article on writing better MCQs. It’s by Mike Dickinson and it’s called Writing Multiple-Choice Questions for Higher-level Thinking. Mr. Dickinson presents several effective techniques for writing better MCQs that I believe you’ll find to be very useful if you use MCQs in your assessments.

Securing Assessment: Don’t ignore your social framework

Posted by Steve Lay

At Questionmark’s European Users Conference in October I took part in a panel discussion on security in assessment. During the session we polled the audience on a number of questions concerning their own security practice. Lots of interesting points came up but one stuck out to me.

International Center for Academic Integrity - Sample Poster - University of California-Davis

Most people in the audience had a formal code that applied to the participant’s conduct during assessments. For an employee engaged in a training program this would be covered by a contract of employment and the organization’s operating manual. For a student at a University it is more likely to be some type of agreement entered into when the student registers. Plagiarism is a particular concern now the web provides such an easy way to share information.

Likewise, most people reported that misconduct on tests would have significant consequences for the participant.

Given these two responses, I was surprised that a minority of the audience used a simple reminder of participants’ obligations before allowing them to proceed with the test.

Most high-stakes testing programmes that I’ve been involved with have had comprehensive rules governing the physical environment used to take tests (whether on paper or on computer). These rules invariably involved placing notices reminding people of the rules of conduct. Of course, most candidates do not stop to read these notices at the time of the exam, but they do help to remind people of the rules they have already signed up to.

As someone involved mainly in the technical side of assessment systems I often hear requests to make the technology stronger against misuse.  More encryption, more control over devices, etc. These solutions play a part, but we often ignore simpler changes to improve detection. It may be hard to remove every opportunity a participant has to cheat, but if you can increase the chances of them getting found out then you can still reduce misconduct. Think of an unlocked room bearing a simple sign “no unauthorized admittance”; add a sign that says the forbidden room is monitored, and the temptation to trespass is reduced.

As someone involved in assessment, I’m mindful of the fact that we all want our participants to succeed and that every case of failure through misconduct is, in some ways, a failure of the system itself. Presenting participants with a brief reminder of their obligations and the fact that their conduct is monitored just before allowing them to proceed with an exam should help persuade the wavering participant that cheating is not in their best interests.

Next Page »