Conference Close-up: Questionmark Boot Camp for Beginners

Posted by Joan Phaup

For people attending the Questionmark Users Conference who are new to our software, we’re pleased to be offering a one-day pre-conference workshop that will get them off to a great start!

Our own Rick Ault, who has led many a Questionmark product training course, will lead Questionmark Boot Camp: Basic Training for Beginners at the Ritz Carlton New Orleans on Tuesday, March 20. He shared some details about this with me a few days ago:

“Boot Camp” sounds a little scary! What will happen during that workshop?

We call it boot camp in the same sense that a military boot camp gives you basic training in order to get more specialized skills. Our boot camp will give new Questionmark users the basic skills and knowledge so that they can get as much as possible out of the conference. We’ll give people a  solid foundation, so that they can seek more specialized information and drill a little deeper during the conference into the areas that interest them most. We want to empower attendees to make the most of their conference.

Rick Ault

Rick Ault

How much do you expect people  to learn in one day?

They’ll get a broad overview of how things work and pick up some skills by actually doing things and getting some practice. We’ll cover basic concepts such as how you create a question, how you group questions into an assessments, then publish it, schedule it, deliver it and get the results back. From start to finish, they’ll bulid up a foundation that will help them have meaningful discussions during the conference. We’re distilling our regular three-day training course into one day, so it’s not nearly as deep, but the scope of the class is the same as the longer course.

What do you hope people will take away from their day in class?

I hope they’ll take away a good understanding of how Questionmark works. I hope they’ll be able to go back to work being able to create some basic assessments, and that they’ll be on their way to doing more advanced things if they decide to go that route.

How should participants prepare for Boot Camp?

Get a good night’s rest, bring your computer and make sure it works!

What are you looking forward to at the conference?

It’s always nice to shepherd my flock – to see my past training attendees and see how they’ve grown and how they’re using Perception today…They teach me new things sometimes!

If you have not signed up for the conference yet, we hope you will sign up in the coming week to save $100 on your registration fee! (Early-bird registration ends next Friday, January 27.)

Conference Close-up: Timing is Everything

Posted by Joan Phaup

As many followers of this blog will know, Questionmark Chairman John Kleeman has been exploring the findings of cognitive learning research and considering how it can apply to assessments.

At this year’s Questionmark Users Conference in New Orleans, John will explain research on the value of spacing out learning and assessment as means of helping people remember information for the long term.

I asked him the other day about his presentation, Timing is Everything : Using psychology research to make your assessments more effective”

What do you value you most about what you’ve been learning from the work of cognitive learning researchers?

John Kleeman

There’s a lot of evidence from cognitive psychology that could make a difference in how we do assessments, training and learning, and I want people to be aware of it. I’m excited about research that’s being done and am keen to share research findings so that people involved in learning and assessment can use that evidence in practical ways.

What key findings about the timing of assessments will you share during your breakout session?

There’s a fascinating and well-documented finding that if you space out learning – separate it out – it’s much more effective than if you do it all in one chunk. Learning should be regarded as a process, not an event. Research shows that if you spend half an hour a day for four days learning something it will be more effective than if you cover all the material in a single two-hour session. Because of the way the mind works, having breaks between learning sessions will help you remember information for the long term. I’ll be sharing solid evidence about this and will talk about its implications.

Assessment plays into all this! An assessment with feedback is learning, and so if you take a series of separated-out assessments, this gives you spacing. Also if you give learners a series of assessments during a course, and encourage them to learn and revise for each assessment, you are encouraging this good behavior of spacing out their learning. So if you have people take quizzes and tests throughout learning instead of just at the end, you are promoting some good learning habits.

Can you give me an example of this?

I wrote not too long ago about how the University of Lund in Sweden uses embedded assessments for knowledge checks within a SharePoint-based learning platform. They’ve found that requiring students to take quizzes as they work through distance learning courses forces them to engage with the material and practice retrieving information from the very start of their courses — and to keep that engagement going throughout the course. That’s a great way to prevent people from cramming a lot of learning into a short period of time – as they might do before a final exam. So assessments can be used to help space out learning. If you just rely on final tests or exams then there is a risk of encouraging people to cram at the end of the course, rather than helping them learn and remember information for the long term.

How can we use research findings about the spacing of learning in designing tests and quizzes?

One of the things I covered last year and will review in this session is the benefit of retrieval practice – the fact that if you want to remember something for the long term then you want to practice retrieving it from memory. Things you have practiced retrieving are easier to remember. So if you have to answer a question or take a test on material, it makes it more likely you will remember it in future. For example, for retrieval practice it’s best if possible to use open-ended questions that require the person to recall information rather than recognize it from a multiple choice list. And you definitely should include feedback.

Another point to keep in mind is that the material that gets tested is the material people will remember, so it’s important to cover the material you really regard as valuable and worth remembering. We’ll discuss these and other ideas about how to apply the research to assessments during the session.

What you do expect participants in your session to take away from it?

I’ll be sharing some of the evidence from cognitive psychology both to communicate to what I understand to be the results and to have people come up with their own views. I will also share links to resources from experts like Dr. Will Thalheimer of Work-Learning Research so that they can learn more on their own.

I don’t just want to say, “Here’s the data and believe me!” I want to give attendees a way to look into this themselves and understand it. Once they grasp the principles coming out of the research, it will help them formulate assessment practices and plans in ways that apply to their individual organizations’ needs.

We hope to see you in New Orleans March 20 – 23 and encourage you to sign up by January 27 for early-bird savings.

Conference Close-up: Using the Net Promoter® Score in Training Evaluations

Joan PhaupPosted by Joan Phaup

Last week I mentioned a peer discussion at the upcoming Questionmark 2012 Users Conference about the relative merits of proctored and non-proctored assessments.

Today I’d like to share a conversation I had recently with Frank Loforte, who will be one of our case study presenters in New Orleans March 20 – 23. Frank works for Beckman Coulter, Inc., where he is involved in training and certifying the company’s technicians and customers in the use of the company’s biomedical testing products, During his presentation, he’ll be describing how his team is using the Net Promoter® Score in training evaluation surveys.

Tell me about your work at Beckman Coulter.

Frank Loforte

Frank Loforte

I do a little bit of everything, but my main job is principal technical trainer. I have been teaching service engineers how to use the instruments our company sells. Along with that I maintain the computers and the websites as well as running tests and surveys. That’s where Questionmark comes in: we use it primarily for our (Kirkpatrick Model) Level 1 and Level 2 assessments.

How are you using Questionmark?

We use it to monitor how the customers and service engineers who come through our training center respond to our courses. We track those surveys and evaluations to make sure they are pleased with the training. We also gauge our students’ levels of knowledge, application and analysis skills. We give a knowledge test, an application test and an analysis test, which we grade separately. Then we report those scores to their managers in the field. We also go back three months after a course to ask managers how well their students are performing, in order to get the supervisor’s point of view as well as the student’s point of view.

You’re going to be talking the Net Promoter Score at the Users Conference. Can you explain a little about that?

We have been using Net Promoter Score questions for about a year and half. This helps us track and quantify how many students are pleased with the training. Many companies use this kind of question for collecting people’s opinions about products. So when they sell a widget they ask customers if they would recommend the widget to someone else on a zero to 10 scale. People who respond with a score of 9 or a 10 are considered promoters. The 8’s and 7’s are called “passives” and those who rate something from 0 to 6 are called “detractors.”

The scoring is kind of complicated (and I’ll be explaining it during my session) but it gives you a really clear indication of how you are doing. Even more valuable is the follow-up question: “What is the most important reason for that score?” We look closely at the responses we get to that question from the detractors and respond to them right away.

What do you expect people to learn from your session?

I’d like them to be able to see what Net Promoter Score is, how we apply it how they can use it in their environments. We have a track record with it now, so I have a lot of data and it’s pretty consistent. It seems we are getting the same type of response from a varied public, so it’s a good indicator. It’s what you do with the answer to the second question that’s really going to be important: fixing the things they say are broken and continuing to do the things they like.

What are you looking forward to at the conference?

I want to learn about other people and what they do. That’s what’s really great: seeing people from all different companies using the same product and asking them questions. I also want to learn more about Questionmark Analytics and any new things that are coming along.

If you would like to learn more about the conference and register online, click here.

Conference agenda nearly full as early-bird registration nears end (Friday!)

Joan PhaupPosted by Joan Phaup

As we zero in on the first early-bird registration deadline for the Questionmark 2012 Users Conference this Friday, I’m pleased to report that the agenda is nearly full.

The line-up may change somewhat between now and March 20 – 23, when we convene at the Ritz Carlton New Orleans, but here’s an outline of what’s in store.

Before the conference gets rolling, there are two optional full-day workshops to choose from on Tuesday, March 20:

During her presentation, Look Before You Leap: What You Measure is What You Get. our keynote speaker, Dr. Jane Bozarth, will include tips on planning assessments with clear objectives and outcomes.

Here’s the current list of breakouts, which you explore in detail within the conference agenda:

Questionmark Features & Functions

  • Introduction to Questionmark Perception for Beginners
  • Answers to Windows Authoring FAQs
  • Using Questionmark Live for Surveys and Course Evaluations
  • Reporting and Analytics: Understanding and sharing assessment results
  • Deploying Questionmark Perception 5.4
  • Questionmark Live Bring-your-own-laptop Tutorial
  • Customizing the Participant Interface

Case Studies

  • An Architect’s Approach to Questionmark Assessment Development: How to Architect, Design and Implement an Efficient Assessment-Building Process
  • How e-Testing is Improving Assessment for the U.S. Coast Guard
  • The Net Promoter Score (NPS) Question Type
  • Analysis of Examination Time Data at the Question- and Person-Specific Level with Perception
  • The Big Switch: Moving Training and Assessment to Mobile Devices
  • Using Questionmark to Answer Business Questions Related to Your Work Processes
  • Installation and Deployment of Perception over the U.S. Navy’s Internet

Best Practices

  • Applying the Principles of Item and Test Analysis to Your Assessment Program
  • Timing is Everything : Using psychology research to make your assessments more effective
  • Alignment, Impact and Measurement with the A-model
  • Instructional Design for the Real World
  • Using Captivate and Flash Simulations in eLearning and Assessments
  • Using Web Services to Integrate with Questionmark Perception


  • Proctored versus non-proctored: How does assessment setting affect student achievement on web-based assessments?
  • Managing Assessment Security as the Stakes are Getting Higher Test Defensibility
  • Using the Angoff Method to Set Cut Scores

Drop-in Demos

  • Delivering Assessments to Mobile Devices
  • What’s New in Perception 5.3 and 5.4?

Register for the conference by December 9th to save $200!

Questions? Email We’re happy to help!

Questionmark Users Conference

Podcast: Jane Bozarth on looking before you leap into learning measurement

Joan PhaupPosted by Joan Phaup

I had a great time the other day chatting with Dr. Jane Bozarth, our keynote speaker for the Questionmark 2012 Users Conference in New Orleans March 20 – 23.

Jane Bozarth

Dr. Jane Bozarth

Jane, whose degrees include a doctorate in Training and Development, will be speaking from her extensive experience as a training practitioner for more than 20 years. She is Elearning Coordinator for the state of North Carolina as well as a book author, Learning Solutions Magazine columnist and blogger.

During her keynote, Look Before You Leap: What You Measure is What You Get, Jane will share methods for building assessment directly into learning design.

We are delighted that she will also present a best practice session on Instructional Design for the Real World,  for those looking for tools and tricks that will support rapid instructional design and get to the heart of needs analysis and improve communication with subject matter experts, managers and others. You can get details about this and other breakout sessions by visiting the conference agenda.

Early-bird conference registration is open until December 9th, so this is a good time to sign up!

Listen in on my conversation with Jane in this podcast, or click here for a transcript.


Keynote for Questionmark 2012 Users Conference: Dr. Jane Bozarth

Joan PhaupPosted by Joan Phaup

With our European counterparts having recently wound down their conference in Brussels, we are revving up for the 2012 Users Conference in New Orleans March 20 – 23!

Jane Bozarth

Dr. Jane Bozarth

Our first major announcement is that Dr. Jane Bozarth, Elearning Coordinator for the State of North Carolina, will be our keynote speaker!

Jane’s talk, Look before You Leap: What You Measure is What You Get, will emphasize the importance of planning quizzes and tests with solid objectives and clear outcomes in mind. She will demonstrate the ways in which assessments can fail to measure the right things, and she will share some tips for improving assessments by building them right into the design of a learning program.

Jane holds a M.Ed. in Training and Development/Technology in Training and a doctorate in Training and Development. She is a popular conference speaker and has written several books including From Analysis to Evaluation: Tools, Tips, and Techniques for Trainers. Jane also writes Learning Solutions Magazine’s popular Nuts and Bolts column.

Online registration is now open, with savings of $200 for those who register by December 9. Group discounts are available, too, so be sure to invite your colleagues to join you!

Experienced Questionmark users please note: The conference call for proposals is open through October 28. Case study and peer discussion proposals are rolling in, and we’d like to receive one from you, too!