Sneak Peek: Conference Agenda & Special Workshop in Santa Fe

Posted by Julie Delazyn

Certified Performance Technologist Jim Parry

Questionmark Conference 2017 is the place to be for vital info and training on the latest assessment technologies and best practices. This event will give you the tools you need to take your assessments to the next level and harness the power of your results.

Register by February 22 — your final chance to save for early-bird discounts.

This year, we’re offering a special pre-conference session. Advanced Test Item Writing Workshop – Learn how to test more than just knowledge. Join Certified Performance Technologist Jim Parry – an expert user of Questionmark technologies – in this fast paced, high-powered workshop.

What’s on the agenda? Here’s a sneak-peek of what you can expect in Santa Fe March 21-24:

  • Explore the Advanced Editor in Next Gen Authoring
  • Avionics: A hands-on, performance-based certification using Questionmark at SpaceTEC
  • Certifying Services-Oriented Associates at PetSmart
  • Setting a Fair, Defensible Cut (Passing) Scores
  • Staying Ahead of Evolving Security Threats

Item writing workshop in San Antonio March 4: Q&A with Mary Lorenz

Joan Phaup 2013 (3)Posted by Joan Phaup

Last week I shared a conversation with Melissa Fein  about her March 4 morning workshop on Test Development Fundamentals in San Antonio, prior to the Questionmark 2014 Users Conference.

Our afternoon workshop that day will give people a chance to drill down into the building blocks of good tests: well-written items. Mary Lorenz, who honed her test writing skills as a program specialist for the Texas State Board for Education Certification and during 11 years as a classroom teacher, will lead this 3-hour session on The Art and Craft of Item Writing.

Participants will learn how to construct well-written multiple choice items that measure course objectives. They’ll also evaluate their own items, identify flaws and polish up their work to achieve more meaningful results.

I asked Mary about her approach to this subject, which she has taught during workshops for ASTD, The University of Texas at Austin and other organizations.

What makes writing good test questions so challenging?


Mary Lorenz

I find that people write test questions from what it is they know, but all too often they have only ever known bad test questions! There are not many people who have training in how to identify and write a good test question.

What are the most common flaws you see in test questions?

A common problem is the lack of a single, clear, correct answer. Another is a poorly written stem that doesn’t provide enough detail. It’s essential to include all of the information necessary to provide a reasonable basis for responding.

Sometimes an author picks a statement out of a text book and use it as a test answer. Materials like that often cue test takers about the correct answer. People have become test-wise and can guess the right answer without really knowing the content. So you have to learn to write better test questions in order to accurately assess knowledge.

What’s the most important question to ask yourself when writing test items?

What is it you are asking test takers to do? It’s not about what information they can remember. It’s what you want them to do with that information. One of the best ways to get learners to think beyond what they “know” is to present them with a situation and ask them what they should do next. Make them apply what they’ve learned to a decision they need to make on the job. Each item should focus on an important concept, typically a common or potentially serious problem or issue related to their work. Don’t waste testing time with questions assessing knowledge of trivial facts. Don’t ask them to simply parrot a definition. Focus on problems they would encounter in real life. Avoid trivial, “tricky,” or overly complex questions.

So it’s really about objectives.

Yes! Test questions flow very easily from solid objectives, but people haven’t been well schooled in how to write a good objective. Designing an assessment, as well as crafting an objective, requires focus. So we will be looking at typical course objectives and comparing those to well-written, assessable objectives.

If you’ve written a good objective, the questions almost write themselves. Your objectives will also help you determine what item type would be most appropriate. I’ll be focusing on multiple choice items during this workshop, but we will touch on how to determine the right item types to use in different contexts.

How do you inject some fun into helping people learn to write good test questions?

First let me admit something: I am an item-writing nerd. Seriously, I have found myself on more than one occasion bordering on giddy when I come up with a novel way of approaching an objective and genuinely frustrated when I have to begin a question with “Which statement is true?”  In that spirit, I show students some classically bad questions and we all have a good laugh over those. I like them to be able to say, “Now that I know some things about how to discern an ‘okay’ question from a ‘good’ question, it’ will be easy for me to avoid writing bad questions.”

I also help people with their own questions, showing them how to make them better. This can be embarrassing at times, but people gain an awareness that they can do better. They understand that it takes effort and it takes time, but it’s worth it to be able to assess what’s really going on.

How should people prepare for this workshop?

I would like them to bring sample questions with them. I would also like them to bring the objectives on which they are trying to base their assessments.

What would you like people will take away from this session?

An awareness of how to do this better. How to take what they already have and make it a more valid and reliable exam. I’d also like them to leave knowing what a good test question looks like. I want them to leave excited about the notion of writing better test questions. I like seeing those light bulbs go off above people’s heads – to see people change their attitude about multiple choice items and discover that, If they’re written well, they can really assess a lot!

In addition to the two half-day workshops, we are offering a full-day Questionmark Boot Camp for Beginners, taught by Questionmark Trainer Rick Ault.

Check out the conference program to see all the educational sessions taking place in San Antonio March 4 – 7.

Register for the conference by January 30th to save $100. 



Questionmark Live POP Quiz!


Posted by Jim Farrell

As I traveled around the US this past fall going to Questionmark Breakfast Briefings and User Group Meetings, it was great fun to introduce participants to our new browser-based authoring tool for subject matter experts, Questionmark Live.

Since then I’ve continued spreading the word about this easy authoring tool to Questionmark Nation, so I figure it’s high time for a pop quiz on the subject! This is a low-stakes assessment, so you can just have some fun answering the questions. It’s also “open book,” so you are welcome to look for answers here!

Good luck on the quiz. Please share it with your friends and co-workers! You can try out Questionmark Live for yourself at Our software support plan customers use it free of charge, but anyone who wants to experiment with it can sample it for free.

If you are using Questionmark and want to see Questionmark Live in action, come to our bring-your-own laptop Questionmark Live  Item Writing Workshop at the Questionmark Users Conference March 14 – 17 in Miami.

Into the third decade of Internet assessment

john_smallPosted by John Kleeman

As I take stock of the past year and look forward to the next, I’m doing so with great appreciation for all of Questionmark’s customers and users and for all my colleagues. I’d like to extend my best wishes to everyone for 2010.

I’m looking forward to the New Year as the start of a decade that promises to be an exciting one in which to work.There should be some amazing new developments during the 2010s as the Internet—which I regard as the third great invention of humankind—continues to transform assessment!

I’d suggest the first great invention was writing, allowing people to store information and knowledge and pass it on to others. Before writing, information could only be transferred by songs and ballads, or one person’s advice to another. Once writing was invented, knowledge could be written and stored, and more complex societies could start to form.

The next great invention was the printing press. Books and knowledge were not just restricted to the rich or the learned but could be communicated to all. Access to information and ideas became much easier, and this led in time to equality and freedom and our modern world.

The Internet, by bringing everyone together irrespective of geography and allowing the synergy of the crowd, rather than just from the writer to the reader, is transforming the world and society.  These are exciting and interesting times to be living and working. We don’t yet know where the Internet will take us, but it already is changing our world and bringing it together.

Working in the field of assessment, we are fortunate to be enablers for the Internet revolution. The Internet means that knowledge, learning and ideas spread faster than ever before, and we are freed from the constraints of geography. You can read this blog entry from Kansas to Karachi, at the time you want. And you can comment on it, tweet it, or ignore it and move to something better, all in just a few keystrokes.

Assessment contributes to the Internet and the Internet contributes to assessment. The Internet is about learning and assessment is the cornerstone of learning: it diagnoses what you know and need to learn, helps confirm what you have learned and helps personalize your learning path.  The promise of the Internet is that a child in the furthest corner of the world can learn from the greatest teacher, and that the potential of everyone in the world can be fulfilled. By effective use of surveys, quizzes, tests and exams, we can be part of making this happening.

The Internet also contributes to making assessment better. From online item writing workshops and item review with tools such as Questionmark Live, to delivery over the Internet and on mobile devices, and to passing and confirming assessment results online, the Internet changes every part of assessment. And this can only change and change for the better as the technology becomes more reliable and the demands for global assessment increase.

The 1990s saw the beginnings of Internet assessment (with Questionmark proud to have produced the first Internet assessment product in 1995) and the 2000s have seen Internet assessment become widespread and useful. Questionmark has an exciting announcement to make about a new version of our software in January 2010 that will make mobile assessments and assessments in frames within wikis, blogs and portals much, much easier. With news like that for starters, the 2010s look a very exciting time for Internet assessment.

Podcast: Using Questionmark Live for Item Writing Workshops

Posted by Joan Phaup

The School of Justice at Miami Dade College in Florida plays an important role in helping people prepare for careers in public safety. Not only does the school offer degree programs, it is also home to the state’s basic abilities testing program for prospective police, correctional and probation officers.

The Florida Basic Abilities Test (F-BAT), which measures basic abilities of recruits planning to enter training programs, needs to be kept current and relevant. The College reviews the test annually and freshens the questions, drawing on the expertise of subject matter experts (SMEs) from all over the state.

The introduction  this year of Questionmark Live has given the college a new, more efficient  way of working with SMEs: During a recently item writing workshop held at the College, SMEs were given a quick tutorial on the use of Questionmark Live and were soon creating questions using its simple browser-based authoring tools. Their efforts were so successful  that the college plans to run future workshops at a distance, doing away with the need for SMEs to meet together in order to help create test content.

You can learn more about the college’s role here and find out more about the item writing sessions by listening to the following podcast with Lebsica Gonzalez, F-BAT project manager at Miami Dade College.