Adding a “question not answered” outcome in Questionmark Live

Posted By Doug Peterson

When people are creating questions, they will very often add an extra outcome that accounts for participants who do not answer a question. This allows the author to give specific feedback for this situation. This video shows you how to add an additional “question not answered” outcome in Questionmark Live!

Announcing two pre-conference workshops in New Orleans March 20, 2012

Joan PhaupPosted by Joan Phaup

Sharon Shrock

Sharon Shrock

As we continue to build the program for the Questionmark 2012 Users Conference we are delighted to announce two full-day pre-conference workshops at the Ritz Carlton New Orleans on Tuesday, March 20, 2012: 

Criterion-Referenced Test Development, with Sharon Shrock and Bill Coscarelli. Sharon and Bill — who have presented outstanding keynote addresses at two previous users conferences  — will help participants in this workshop understand testing best practices, meet rigorous competency testing standards and interpret test results correctly.

The workshop,  based on Sharon and Bill’s book, Criterion-Referenced Test Development: Technical and Legal Guidelines for Corporate Training, is ideal for trainers, instructional designers, course developers and training managers. Understanding the principles of skillful test authoring will help you create ethical and dependable testing programs that yield meaningful, measurable results.

Bill Coscarelli

Bill Coscarelli

Rick Ault

Rick Ault

Here’s a great way to learn the basics of using Questionmark technologies before the conference starts: Bring your laptop to a basic training course! Questionmark Trainer Rick Ault will give you hands-on practice  in creating questions, putting together an assessment, then scheduling it, taking it and seeing the results. If you are just starting out — or if you have little experience with Questionmark — joining this workshop will give you a head start and help you get the most from the conference program.

Whether you add a workshop to your plans or not, now is a great time to register for the conference: Early-bird registration savings of $200 are available through December 9th.

Using QR codes to direct smartphone users to assessments

QR or “Quick Response” codes offer an easy way to direct users of smartphones, tablets and other mobile devices to your assessments — and thereby to boost participation in surveys, course evaluations, knowledge checks, quizzes and other assessments.Android phone

There are various ways to create QR codes for Questionmark assessments, but here are the basics: 

  • Create your assessment and generate a URL for it
  • Provide your assessment URL to a service such as, orangeqr or to generate a short URL for the assessment
  • Copy the short URL in to your browser address bar and append .qr to the end of it
  • Navigate to that page, where you will see a QR code for you to save for future use

See these PowerPoints on SlideShare for more details about using QR codes to launch Questionmark assessments.

Measuring learning in SharePoint: where to find info

Posted by Julie Delazyn

The way we learn is changing. By allowing us to more easily share information and acquire knowledge, the Internet has made it easier to learn informally. Moving away from the traditional academic model, we are increasingly learning from each other and on the job.

Microsoft SharePoint’s popularity as a collaboration environment for everyday work tasks makes it a readily available environment for learning functions — an idea that fits in well with the 70+20+10 learning model. Assessments also fit in well with that model, and with SharePoint, too.

Many types of assessments can work well with SharePoint – everything from quizzes, diagnostic tests, knowledge checks and competency tests to surveys and course evaluations. No matter what the setting – a formal learning program, regulatory compliance, performance support or an employee/partner portal, perhaps – assessments have key roles to play.

How to include assessments in SharePoint?
•    Inbuilt SharePoint – functional for basic surveys
•    Custom web parts – write your own!
•    Embed Flash apps – possible for simple quizzes
•    Embed web apps – easy to do. (See how a Questionmark user has embedded a quiz to engage learners.)

If you would like to learn more about using assessments within SharePoint, you can check out this Questionmark presentation on SlideShare.

For more details, download the white paper Learning and Assessment on SharePoint or visit John Kleeman’s SharePoint and Assessment blog.

Problem, Performance and Program in the A-model

Posted by John Kleeman

How do you evaluate the effectiveness of training?The four levels of the Kirkpatrick Model

Around 50 years ago, Dr. Donald Kirkpatrick wrote about what became known as the Kirkpatrick model – where you evaluate training in four levels: reaction, learning, behavior and results. His model has been hugely influential in the evaluation of workplace learning and the use of assessment.

This year, Dr Bruce C. Aaron, whose career spans the Florida Department of Education and working in evaluation at Accenture as well as volunteering at the chapter and national level with ASTD, has written about the A-model, a new way of thinking about assessment and evaluation of workplace learning.

The A-model does not start with the question: “How do you evaluate the effectiveness of training?” Instead, it puts the training in context, by starting with the 3 P’s: Problem, Performance and Program.

The basic concept is that in order to make sense of evaluating something, you have to start with the business Problem that you are trying to solve. Problems need to be important for the business purpose and they must be measurable – for example a need for increased production, improved quality, revenue increase, higher customer satisfaction.

Once you have a clear definition of the Problem (and how you will measure if it is solved), you need to define the human Performance that will solve the Problem. Performance itself will typically be behavior that is directly tied to important accomplishment or results in the workplace. To get performance, you use Performance Enablers – new learning of knowledge, new skills, changed attitude, feedback or incentives, performance support or new equipment. For example, you might identify that people need to be more skilled with a computer program to solve the Problem, or else to have better product knowledge.

Only after defining the Problem and the Performance, should you define the Program. The Program consists of the intervention or solution to the problem, and might be a training course, some other kind of learning or improvement or performance support.

Just as in the picture below, the Program is just the tip of the iceberg, underpinned by the Performance and the Problem. Without them, evaluation of the Program is not meaningful or useful.

The 3Ps in the A-model

Thus if your Program is a training course or learning intervention, you evaluate its effectiveness by measuring not only the delivery of Program itself but also determining whether it achieves the human performance improvement you are looking for and has a positive impact on the Problem it is there to solve.

A key difference between the Kirkpatrick model and the A-model is that the Kirkpatrick model starts out with the training course and goes on to tie it into results, whereas the A-model starts with a business problem and helps you define how to improve performance to solve the problem, and then measure training or other performance support in that context.

In my next post on this topic, I’ll talk about how the A-model informs the processes of analysis, design and evaluation and where assessments fit in. In the meantime, you may want to listen to this 8-minute podcast from Dr. Aaron or read his thought-provoking A-model white paper available here (free with registration).

Looking forward to the UK Academic Briefing

There’s just one week to go until the UK Academic Briefing at the University of Central Lancashire. I’ve just finished arranging my travel, and I’m looking forward to meeting everyone there.

As many of you will already know, I worked for many years in the UK academic community working on computer based learning projects.

One project I was often asked about when I was working at the University of Cambridge was my work on the educational applications of the Digital Desk, as reported in ALT-J way back 1997. The Digital Desk used an overhead camera to automatically recognize paper documents placed on it, enabling interaction with them. The system used a sort of printed bar-code and required a seriously expensive camera to function. In that paper I cautiously warned that “it will be some time before computer-augmented environments make their way onto the average laboratory desktop.”

But 15 years later and this technology is embedded in most mobile phones and devices. The bar-code technology has improved and is now the recognizable QR codes you’ll see printed in magazines and on posters. I’m pleased to see that the future has now arrived: using QR codes to launch Questionmark assessments is just one of the topics we’ll be covering at the briefing.

I’ll be there to answer any questions you have for the Product Management team in addition to taking you through what’s new in the latest Blackboard Connector and giving you an update on what we’re currently working on.

I’ll be keeping an eye on the back channel so feel free to post your questions and feedback to #qmcon.

If you haven’t already registered for this free learning event that will take place on November 25 at the University of Central Lancashire, click here for details and online registration.