Posted by Joan Phaup
Last week I shared a conversation with Melissa Fein about her March 4 morning workshop on Test Development Fundamentals in San Antonio, prior to the Questionmark 2014 Users Conference.
Our afternoon workshop that day will give people a chance to drill down into the building blocks of good tests: well-written items. Mary Lorenz, who honed her test writing skills as a program specialist for the Texas State Board for Education Certification and during 11 years as a classroom teacher, will lead this 3-hour session on The Art and Craft of Item Writing.
Participants will learn how to construct well-written multiple choice items that measure course objectives. They’ll also evaluate their own items, identify flaws and polish up their work to achieve more meaningful results.
I asked Mary about her approach to this subject, which she has taught during workshops for ASTD, The University of Texas at Austin and other organizations.
What makes writing good test questions so challenging?
I find that people write test questions from what it is they know, but all too often they have only ever known bad test questions! There are not many people who have training in how to identify and write a good test question.
What are the most common flaws you see in test questions?
A common problem is the lack of a single, clear, correct answer. Another is a poorly written stem that doesn’t provide enough detail. It’s essential to include all of the information necessary to provide a reasonable basis for responding.
Sometimes an author picks a statement out of a text book and use it as a test answer. Materials like that often cue test takers about the correct answer. People have become test-wise and can guess the right answer without really knowing the content. So you have to learn to write better test questions in order to accurately assess knowledge.
What’s the most important question to ask yourself when writing test items?
What is it you are asking test takers to do? It’s not about what information they can remember. It’s what you want them to do with that information. One of the best ways to get learners to think beyond what they “know” is to present them with a situation and ask them what they should do next. Make them apply what they’ve learned to a decision they need to make on the job. Each item should focus on an important concept, typically a common or potentially serious problem or issue related to their work. Don’t waste testing time with questions assessing knowledge of trivial facts. Don’t ask them to simply parrot a definition. Focus on problems they would encounter in real life. Avoid trivial, “tricky,” or overly complex questions.
So it’s really about objectives.
Yes! Test questions flow very easily from solid objectives, but people haven’t been well schooled in how to write a good objective. Designing an assessment, as well as crafting an objective, requires focus. So we will be looking at typical course objectives and comparing those to well-written, assessable objectives.
If you’ve written a good objective, the questions almost write themselves. Your objectives will also help you determine what item type would be most appropriate. I’ll be focusing on multiple choice items during this workshop, but we will touch on how to determine the right item types to use in different contexts.
How do you inject some fun into helping people learn to write good test questions?
First let me admit something: I am an item-writing nerd. Seriously, I have found myself on more than one occasion bordering on giddy when I come up with a novel way of approaching an objective and genuinely frustrated when I have to begin a question with “Which statement is true?” In that spirit, I show students some classically bad questions and we all have a good laugh over those. I like them to be able to say, “Now that I know some things about how to discern an ‘okay’ question from a ‘good’ question, it’ will be easy for me to avoid writing bad questions.”
I also help people with their own questions, showing them how to make them better. This can be embarrassing at times, but people gain an awareness that they can do better. They understand that it takes effort and it takes time, but it’s worth it to be able to assess what’s really going on.
How should people prepare for this workshop?
I would like them to bring sample questions with them. I would also like them to bring the objectives on which they are trying to base their assessments.
What would you like people will take away from this session?
An awareness of how to do this better. How to take what they already have and make it a more valid and reliable exam. I’d also like them to leave knowing what a good test question looks like. I want them to leave excited about the notion of writing better test questions. I like seeing those light bulbs go off above people’s heads – to see people change their attitude about multiple choice items and discover that, If they’re written well, they can really assess a lot!
In addition to the two half-day workshops, we are offering a full-day Questionmark Boot Camp for Beginners, taught by Questionmark Trainer Rick Ault.
Check out the conference program to see all the educational sessions taking place in San Antonio March 4 – 7.
Register for the conference by January 30th to save $100.