Item writing workshop in San Antonio March 4: Q&A with Mary Lorenz

Joan Phaup 2013 (3)Posted by Joan Phaup

Last week I shared a conversation with Melissa Fein  about her March 4 morning workshop on Test Development Fundamentals in San Antonio, prior to the Questionmark 2014 Users Conference.

Our afternoon workshop that day will give people a chance to drill down into the building blocks of good tests: well-written items. Mary Lorenz, who honed her test writing skills as a program specialist for the Texas State Board for Education Certification and during 11 years as a classroom teacher, will lead this 3-hour session on The Art and Craft of Item Writing.

Participants will learn how to construct well-written multiple choice items that measure course objectives. They’ll also evaluate their own items, identify flaws and polish up their work to achieve more meaningful results.

I asked Mary about her approach to this subject, which she has taught during workshops for ASTD, The University of Texas at Austin and other organizations.

What makes writing good test questions so challenging?


Mary Lorenz

I find that people write test questions from what it is they know, but all too often they have only ever known bad test questions! There are not many people who have training in how to identify and write a good test question.

What are the most common flaws you see in test questions?

A common problem is the lack of a single, clear, correct answer. Another is a poorly written stem that doesn’t provide enough detail. It’s essential to include all of the information necessary to provide a reasonable basis for responding.

Sometimes an author picks a statement out of a text book and use it as a test answer. Materials like that often cue test takers about the correct answer. People have become test-wise and can guess the right answer without really knowing the content. So you have to learn to write better test questions in order to accurately assess knowledge.

What’s the most important question to ask yourself when writing test items?

What is it you are asking test takers to do? It’s not about what information they can remember. It’s what you want them to do with that information. One of the best ways to get learners to think beyond what they “know” is to present them with a situation and ask them what they should do next. Make them apply what they’ve learned to a decision they need to make on the job. Each item should focus on an important concept, typically a common or potentially serious problem or issue related to their work. Don’t waste testing time with questions assessing knowledge of trivial facts. Don’t ask them to simply parrot a definition. Focus on problems they would encounter in real life. Avoid trivial, “tricky,” or overly complex questions.

So it’s really about objectives.

Yes! Test questions flow very easily from solid objectives, but people haven’t been well schooled in how to write a good objective. Designing an assessment, as well as crafting an objective, requires focus. So we will be looking at typical course objectives and comparing those to well-written, assessable objectives.

If you’ve written a good objective, the questions almost write themselves. Your objectives will also help you determine what item type would be most appropriate. I’ll be focusing on multiple choice items during this workshop, but we will touch on how to determine the right item types to use in different contexts.

How do you inject some fun into helping people learn to write good test questions?

First let me admit something: I am an item-writing nerd. Seriously, I have found myself on more than one occasion bordering on giddy when I come up with a novel way of approaching an objective and genuinely frustrated when I have to begin a question with “Which statement is true?”  In that spirit, I show students some classically bad questions and we all have a good laugh over those. I like them to be able to say, “Now that I know some things about how to discern an ‘okay’ question from a ‘good’ question, it’ will be easy for me to avoid writing bad questions.”

I also help people with their own questions, showing them how to make them better. This can be embarrassing at times, but people gain an awareness that they can do better. They understand that it takes effort and it takes time, but it’s worth it to be able to assess what’s really going on.

How should people prepare for this workshop?

I would like them to bring sample questions with them. I would also like them to bring the objectives on which they are trying to base their assessments.

What would you like people will take away from this session?

An awareness of how to do this better. How to take what they already have and make it a more valid and reliable exam. I’d also like them to leave knowing what a good test question looks like. I want them to leave excited about the notion of writing better test questions. I like seeing those light bulbs go off above people’s heads – to see people change their attitude about multiple choice items and discover that, If they’re written well, they can really assess a lot!

In addition to the two half-day workshops, we are offering a full-day Questionmark Boot Camp for Beginners, taught by Questionmark Trainer Rick Ault.

Check out the conference program to see all the educational sessions taking place in San Antonio March 4 – 7.

Register for the conference by January 30th to save $100. 



Boot Camp for Beginners set for March 4, 2014

Joan Phaup 2013 (3)Posted by Joan Phaup

New customers who attended the Questionmark Users Conference in the past used to tell us that some hands-on instruction before the start of the conference would help them get a lot more out of the proceedings.

Enter Questionmark Boot Camp: Basic Training for Beginners – where people learn the basics before they join the throng at the conference. This full-day workshop has become a popular pre-conference option,  and we’re bringing it back again on Tuesday, March 4, 2014 – right before the Questionmark 2014 Users Conference in San Antonio.

I don’t get  to attend Boot Camp, but I did spend a few minutes talking about it with our trainer, Rick Ault, who has as much fun there as his pupils:

Rick Ault

Rick Ault

What happens at Boot Camp?
We talk about all the different Questionmark tools — how they are used together — and give people a solid understanding of the process and what they can do to build meaningful assessments. It’s hands-on. We ask people to bring their own laptops so that we can give them actual practice using the software. They have the chance to use Questionmark Live to build questions and assessments, then put that content onto a server to see it work.

Who should attend?
Any new users of Questionmark would benefit from it, because it’s designed to give people an understanding of what the product does and how it works.

What should they bring?
They should bring their laptops, plus some ideas for how they might like to use Questionmark. They should also bring some ideas for some fun content that they might want to create?

How does Boot Camp prepare new customers for the Users Conference?
It gives them exposure to all of the tools, and it helps them understand the process. By getting some hands-on experience with our technologies, they will be able to make better choices about what conference tracks and sessions to attend. They’ll also be able to think of meaningful questions to ask at the conference.

What do YOU like best about Boot Camp?
I like meeting new customers, and I like seeing their happiness when they create something. It’s great to see the birth of their new content as they join the Questionmark family!

Newcomers to Questionmark can join the familhy in style by attending Boot Camp. You can sign up when you register for the conference.

Two great workshops: Questionmark Boot Camp and Criterion-Referenced Test Development

Rick Ault

Posted by Joan Phaup

Planning for the Questionmark 2013 Users Conference in Baltimore, Maryland, March 3 – 6 is well underway.

We have begun posting descriptions of breakout sessions and are pleased to announce the return — by popular demand — of two pre-conference workshops.

Both of these all-day sessions will take place on Sunday March 3, 2013:

Questionmark Boot Camp: Basic Training for Beginners, with Questionmark Trainer Rick Ault

Learn the basics of using Questionmark technologies before the conference starts. Beginning Questionmark users are invited to bring their laptops to a basic training course.

Get into gear with hands-on practice in creating questions, putting together an assessment, then scheduling it, taking it and seeing the results. Start off with some first-hand experience that will give you a firm footing for learning more at the conference.

sharon shrock

Criterion-Referenced Test Development, with Sharon Shrock and Bill Coscarelli

Bill Coscerelli

Sharon and Bill are the authors of Criterion-Referenced Test Development: Technical and Legal Guidelines for Corporate Training. Participants in their pre-conference workshop, which is based on the book, will explore testing best practices and will learn how to meet rigorous competency testing standards and interpret test results correctly.

This workshop is ideal for trainers, instructional designers, course developers and training managers. Understanding the principles of skillful test authoring will help you create ethical and dependable testing programs that yield meaningful, measurable results.

You can save $200 by registering for the conference on or before November 16th. You can sign up for a workshop at the same time or add in a workshop later. It’s up to you!

Conference Close-up: Questionmark Boot Camp for Beginners

Posted by Joan Phaup

For people attending the Questionmark Users Conference who are new to our software, we’re pleased to be offering a one-day pre-conference workshop that will get them off to a great start!

Our own Rick Ault, who has led many a Questionmark product training course, will lead Questionmark Boot Camp: Basic Training for Beginners at the Ritz Carlton New Orleans on Tuesday, March 20. He shared some details about this with me a few days ago:

“Boot Camp” sounds a little scary! What will happen during that workshop?

We call it boot camp in the same sense that a military boot camp gives you basic training in order to get more specialized skills. Our boot camp will give new Questionmark users the basic skills and knowledge so that they can get as much as possible out of the conference. We’ll give people a  solid foundation, so that they can seek more specialized information and drill a little deeper during the conference into the areas that interest them most. We want to empower attendees to make the most of their conference.

Rick Ault

Rick Ault

How much do you expect people  to learn in one day?

They’ll get a broad overview of how things work and pick up some skills by actually doing things and getting some practice. We’ll cover basic concepts such as how you create a question, how you group questions into an assessments, then publish it, schedule it, deliver it and get the results back. From start to finish, they’ll bulid up a foundation that will help them have meaningful discussions during the conference. We’re distilling our regular three-day training course into one day, so it’s not nearly as deep, but the scope of the class is the same as the longer course.

What do you hope people will take away from their day in class?

I hope they’ll take away a good understanding of how Questionmark works. I hope they’ll be able to go back to work being able to create some basic assessments, and that they’ll be on their way to doing more advanced things if they decide to go that route.

How should participants prepare for Boot Camp?

Get a good night’s rest, bring your computer and make sure it works!

What are you looking forward to at the conference?

It’s always nice to shepherd my flock – to see my past training attendees and see how they’ve grown and how they’re using Perception today…They teach me new things sometimes!

If you have not signed up for the conference yet, we hope you will sign up in the coming week to save $100 on your registration fee! (Early-bird registration ends next Friday, January 27.)

Announcing two pre-conference workshops in New Orleans March 20, 2012

Joan PhaupPosted by Joan Phaup

Sharon Shrock

Sharon Shrock

As we continue to build the program for the Questionmark 2012 Users Conference we are delighted to announce two full-day pre-conference workshops at the Ritz Carlton New Orleans on Tuesday, March 20, 2012: 

Criterion-Referenced Test Development, with Sharon Shrock and Bill Coscarelli. Sharon and Bill — who have presented outstanding keynote addresses at two previous users conferences  — will help participants in this workshop understand testing best practices, meet rigorous competency testing standards and interpret test results correctly.

The workshop,  based on Sharon and Bill’s book, Criterion-Referenced Test Development: Technical and Legal Guidelines for Corporate Training, is ideal for trainers, instructional designers, course developers and training managers. Understanding the principles of skillful test authoring will help you create ethical and dependable testing programs that yield meaningful, measurable results.

Bill Coscarelli

Bill Coscarelli

Rick Ault

Rick Ault

Here’s a great way to learn the basics of using Questionmark technologies before the conference starts: Bring your laptop to a basic training course! Questionmark Trainer Rick Ault will give you hands-on practice  in creating questions, putting together an assessment, then scheduling it, taking it and seeing the results. If you are just starting out — or if you have little experience with Questionmark — joining this workshop will give you a head start and help you get the most from the conference program.

Whether you add a workshop to your plans or not, now is a great time to register for the conference: Early-bird registration savings of $200 are available through December 9th.