Next Generation Assessment Technology & Exciting Events Driving the Conference Agenda

Now that we have the program in place for Questionmark Conference 2017, I’m eager to highlight a few sessions that you will have a chance to attend in Santa Fe, New Mexico March 21-24.

Before the conference gets rolling, there are two full-day workshops available Tuesday, March 21:

Here’s a peak and the agenda. You can explore the entire list of sessions here: Conference Program.

Questionmark Features & Functions

Case Studies

Best Practices

Networking Events

We have some fantastic networking events planned as well.

We’re kicking off the conference with our signature dessert reception. The next day, you will have a chance to enjoy Santa Fe enchanting downtown and dine with a group of fellow assessment professionals. But it’s all culminating to our final event: Meow Wolf’s House of Eternal Return – an exciting multimedia immersive art exhibit experience. Watch out for surprise acts! We look forward to an evening of eating, networking and celebrating!

Questions? Email We’re happy to help!

Item Writing & Questionmark Boot Camp: Pre-Conference Workshops

Rick Ault, Questionmark Trainer

Julie ProfilePosted by Julie Delazyn

Planning for Questionmark Conference 2017 in Santa Fe, New Mexico, March 21-14 is well underway.

We have begun posting descriptions of breakout sessions and are pleased to announce two pre-conference workshops.

Both of these all-day sessions will take place on Tuesday, March 21, 2017:

assessments-2017Questionmark Boot Camp: Basic Training for Beginners

Do you want to get the most out of the Questionmark Users Conference even though you are just starting out with Questionmark?

Here’s what to do: Bring your laptop and learn directly from Questionmark expert Rick Ault.

Bring your laptop and get into gear with hands-on practice in creating questions, putting together an assessment, then scheduling it, taking it and seeing the results. Start off with some first-hand experience that will give you a firm footing for learning more at the conference.


Jim Parry, Owner and Chief Executive Manager of Compass Consultants, LLC

features-functions-2017Advanced Test Item Writing Workshop: Learn how to test more than just knowledge

Writing test items is difficult, but trying to make them check more than knowledge is a huge challenge.

Join Certified Performance Technologist Jim Parry  — an expert user of Questionmark technologies — in this fast-paced, high-powered workshop, which will present a review of the basics of testing and provide hands-on practice to help you turn low complexity, knowledge-based test items into higher complexity, performance-based items following Bloom’s Taxonomy and Gagne’s Nine Events of Instruction.

Conference Registration Tuition:

  • You can save $200 by registering for the conference on or before January 18. You can sign up for a workshop at the same time or add in a workshop later. It’s up to you! Pre-conference workshop or bootcamp add-ons available during the registration process.

Making your Assessment Valid: 5 Tips from Miami

John Kleeman Headshot

Posted by John Kleeman

A key reason people use Questionmark’s assessment management system is that it helps you make more valid assessments. To remind you, a valid assessment is one that genuinely measures what it is supposed to measure. Having an effective process to ensure your assessments are valid, reliable and trustable was an important topic at Questionmark Conference 2016 in Miami last week. Here is some advice I heard:

Reporting back from 3 days of learning and networking at Questionmark Conference 2016 in Miami

Tip 1: Everything starts from the purpose of your assessment. Define this clearly and document it well. A purpose that is not well defined or that does not align with the needs of your organization will result in a poor test. It is useful to have a formal process to kick off  a new assessment to ensure the purpose is defined clearly and is aligned with business needs.

Tip 2: A Job Task Analysis survey is a great way of defining the topics/objectives for new-hire training assessments. One presenter at the conference sent out a survey to the top performing 50 percent of employees in a job role and asked questions on a series of potential job tasks. For each job task, he asked how difficult it is (complexity), how important it is (priority) and how often it is done (frequency). He then used the survey results to define the structure of knowledge assessments for new hires to ensure they aligned with needed job skills.

Tip 3: The best way to ensure that a workplace assessment starts and remains valid is continual involvement with Subject Matter Experts (SMEs). They help you ensure that the content of the assessment matches the content needed for the job and ensure this stays the case as the job changes. It’s worth investing in training your SMEs in item writing and item review. Foster a collaborative environment and build their confidence.

Tip 4: Allow your participants (test-takers) to feed back into the process. This will give you useful feedback to improve the questions and the validity of the assessment. It’s also an important part of being transparent and open in your assessment programme, which is useful because people are less likely to cheat if they feel that the process is well-intentioned. They are also less likely to complain about the results being unfair. For example it’s useful to write an internal blog explaining why and how you create the assessments and encourage feedback.

Lunch with a view at Questionmark Conference 2016 in Miami

Tip 5: As the item bank grows and as your assessment programme becomes more successful, make sure to manage the item bank and review items. Retire items that are no longer relevant or when they have been overexposed. This keeps the item bank useful, accurate and valid.

There was lots more at the conference – excitement that Questionmark NextGen authoring is finally here, a live demo of our new easy to use Printing and Scanning solution … and having lunch on the hotel terrace in the beautiful Miami spring sunshine – with Questionmark branded sunglasses to keep cool.

There was a lot of buzz at the conference about documenting your assessment decisions and making sure your assessments validly measure job competence. There is increasing understanding that assessment is a process not a project, and also that to be used to measure competence or to select for a job role, an assessment must cover all important job tasks.

I hope these tips on making assessments valid are helpful. Click here for more information on Questionmark’s assessment management system.

Item writing workshop in San Antonio March 4: Q&A with Mary Lorenz

Joan Phaup 2013 (3)Posted by Joan Phaup

Last week I shared a conversation with Melissa Fein  about her March 4 morning workshop on Test Development Fundamentals in San Antonio, prior to the Questionmark 2014 Users Conference.

Our afternoon workshop that day will give people a chance to drill down into the building blocks of good tests: well-written items. Mary Lorenz, who honed her test writing skills as a program specialist for the Texas State Board for Education Certification and during 11 years as a classroom teacher, will lead this 3-hour session on The Art and Craft of Item Writing.

Participants will learn how to construct well-written multiple choice items that measure course objectives. They’ll also evaluate their own items, identify flaws and polish up their work to achieve more meaningful results.

I asked Mary about her approach to this subject, which she has taught during workshops for ASTD, The University of Texas at Austin and other organizations.

What makes writing good test questions so challenging?


Mary Lorenz

I find that people write test questions from what it is they know, but all too often they have only ever known bad test questions! There are not many people who have training in how to identify and write a good test question.

What are the most common flaws you see in test questions?

A common problem is the lack of a single, clear, correct answer. Another is a poorly written stem that doesn’t provide enough detail. It’s essential to include all of the information necessary to provide a reasonable basis for responding.

Sometimes an author picks a statement out of a text book and use it as a test answer. Materials like that often cue test takers about the correct answer. People have become test-wise and can guess the right answer without really knowing the content. So you have to learn to write better test questions in order to accurately assess knowledge.

What’s the most important question to ask yourself when writing test items?

What is it you are asking test takers to do? It’s not about what information they can remember. It’s what you want them to do with that information. One of the best ways to get learners to think beyond what they “know” is to present them with a situation and ask them what they should do next. Make them apply what they’ve learned to a decision they need to make on the job. Each item should focus on an important concept, typically a common or potentially serious problem or issue related to their work. Don’t waste testing time with questions assessing knowledge of trivial facts. Don’t ask them to simply parrot a definition. Focus on problems they would encounter in real life. Avoid trivial, “tricky,” or overly complex questions.

So it’s really about objectives.

Yes! Test questions flow very easily from solid objectives, but people haven’t been well schooled in how to write a good objective. Designing an assessment, as well as crafting an objective, requires focus. So we will be looking at typical course objectives and comparing those to well-written, assessable objectives.

If you’ve written a good objective, the questions almost write themselves. Your objectives will also help you determine what item type would be most appropriate. I’ll be focusing on multiple choice items during this workshop, but we will touch on how to determine the right item types to use in different contexts.

How do you inject some fun into helping people learn to write good test questions?

First let me admit something: I am an item-writing nerd. Seriously, I have found myself on more than one occasion bordering on giddy when I come up with a novel way of approaching an objective and genuinely frustrated when I have to begin a question with “Which statement is true?”  In that spirit, I show students some classically bad questions and we all have a good laugh over those. I like them to be able to say, “Now that I know some things about how to discern an ‘okay’ question from a ‘good’ question, it’ will be easy for me to avoid writing bad questions.”

I also help people with their own questions, showing them how to make them better. This can be embarrassing at times, but people gain an awareness that they can do better. They understand that it takes effort and it takes time, but it’s worth it to be able to assess what’s really going on.

How should people prepare for this workshop?

I would like them to bring sample questions with them. I would also like them to bring the objectives on which they are trying to base their assessments.

What would you like people will take away from this session?

An awareness of how to do this better. How to take what they already have and make it a more valid and reliable exam. I’d also like them to leave knowing what a good test question looks like. I want them to leave excited about the notion of writing better test questions. I like seeing those light bulbs go off above people’s heads – to see people change their attitude about multiple choice items and discover that, If they’re written well, they can really assess a lot!

In addition to the two half-day workshops, we are offering a full-day Questionmark Boot Camp for Beginners, taught by Questionmark Trainer Rick Ault.

Check out the conference program to see all the educational sessions taking place in San Antonio March 4 – 7.

Register for the conference by January 30th to save $100. 



Early-bird savings on conference registration end today: Sign up now!

Joan Phaup 2013 (3)Posted by Joan Phaup

Just a reminder that you can save $200 if you register today for the Questionmark 2014 Users Conference.

We look forward to seeing you March 4 – 7 in San Antonio, Texas, for three intensive days of learning and networking.

Check out the conference program as it continues to take shape, and sign up today!

This conference truly is the best place to learn about our technologies, improve your assessments and discuss best practices with Questionmark staff, industry experts and your colleagues. But don’t take my word for it. Let these attendees at the 2013 tell you what they think:



The impact of feedback on learning and retention

Joan Phaup HeadshotPosted by Joan Phaup

Each year at the Questionmark Users Conference we like to include at least one breakout session relating to cognitive learning research – and 2013 is no exception.

John Kleeman, Questionmark’s founder and chairman, takes a special interest in learning research and has been focusing lately on the role feedback plays in improving the value of quizzes and tests.

John will lead a best practices session when we meet in Baltimore March 3 – 6, on Assessment Feedback – What Can We Learn from Psychology Research?

I spent a few minutes asking John about his presentation.

John Kleeman portrait

John Kleeman

What research have you been following on the effects of feedback on learning and retention?

My main role is as chairman of Questionmark, but I keep an active eye on relevant research, and I follow a number of researchers who are looking into how learning and retention work – and I’m particularly interested in how assessments fit into that. For example, I’ve been following Professor Roddy Roediger at Washington University in St Louis, Missouri and several of his colleagues across the U.S.

(Click here to read one of John’s interviews with Professor Roediger.)

What would you say are the key findings from this research?

What we kind of know but don’t always put into practice is that we forget a surprising amount of what we learn. People know about the forgetting curve as an idea, but don’t always think it applies to them! We think we are going to be better at remembering things later than we actually are. A quiz or test can force you into practicing retrieving and that makes it more likely for things to stick in your mind.

When you learn something – whether in a formal or informal context– you won’t remember a lot of it in a month or six months. Taking a quiz or test helps you retain that learning by providing retrieval practice and slowing the forgetting curve. If people take quizzes or tests, it slows down the forgetting curve – and quizzes with feedback slow down the forgetting curve more effectively than quizzes without feedback.

Will you discuss topic feedback and well as question feedback?

A lot of the research covers question feedback because it’s very easy to measure how well people do on a specific fact. But there is also evidence about topic feedback, and yes; I will be covering topic feedback as well as question feedback.

What would you like your audience to take away from your session?

I aim to practice what I preach, so I will use interactive techniques to help people remember what I talk about! I don’t want just to provide theory: I also want to give actionable ideas that people can apply to their Questionmark assessments to improve retention.

I’d like to add that I’ve found from talking with customers that the conference is a fantastic place to learn. People who come to the conference get a lot of formal learning – for instance by presentations from assessment experts and Questionmark staff who explain effective ways to use our technologies – but they also get a lot of informal learning from interacting with other users. I’m especially looking forward to presentations from our expert customers. Some of our case study presenters have been using our software for many years and have a lot of experience and wisdom to share. So I think I’ll learn a lot from those presentations myself!

You can save $100 if you register for the conference by January 18th. Check out the conference agenda and sign up soon!