Writing high-complexity items for performance-based tests
Posted by Joan Phaup
Performance-based tests require complex items that require participants to show that they know how to apply what they have learned.
James Parry from the U.S. Coast Guard will lead a discussion about the challenges involved in writing these types of questions during the Questionmark 2013 Users Conference in Baltimore March 3 – 6: Advanced Question Writing: Creating High-Complexity Test Items.
I asked James for some background on the subject, which is a major interest of his work as Test Development/E-Testing Manager for Coast Guard Training Center Yorktown, Virginia.
Why are higher-complexity test items important?
The push in today’s society is towards performance-based testing to ensure a candidate can actually ‘do’ the job and not just recite knowledge about the job. True performance-based testing involves each individual performing each task while being observed. With our challenging global economy, having both the time and real equipment for each student to perform each task is, most times, cost prohibitive. Well-written questions that address performance rather than just knowledge can help bridge that gap.
Why is it more difficult to write questions that go beyond checking someone’s knowledge?
Performance-based test items are expensive and time consuming to develop compared to knowledge-based items. A solution is a hybrid approach which can use a traditional objective-based written test item to check the foundation knowledge at the highest cognitive level possible as a precursor to the actual performance test. Being able to develop test items at a higher cognitive level is difficult because the designer has to think outside of the traditional multiple choice test item and more towards the performance.
What’s your own experience writing higher-complexity test items?
The Coast Guard requires performance-based testing at all of its entry-level schools to ensure the young men and women going into the field can actually perform the jobs they are trained to do. As the test development manager I work with the test authors at the schools to help build foundational assessments at various levels of complexity to test the students comprehension as close to the actual performance as possible before actually performing the tasks.
What will be happening during your discussion session?
I will be discussing how both Bloom’s Taxonomy and Gangé’s nine Instructional events relate to both complexity and cognitive levels of testing. I’ll share an example of how a simple objective can be tested at just about any level of complexity and couple the example with Bloom’s Taxonomy. Then I’ll ask the participants to develop a test item at all of the complexity and taxonomy levels.
Who do you think will be benefit most from this discussion?
Participants who have a basic understanding of test item development will benefit the most, but novice test developers and training managers will benefit as well.
How can people prepare for this session?
Those who attend Sharon Shrock and Bill Coscarelli’s pre-conference workshop on Criterion-Reference Test Development will have a good foundation to build on – but just reading up on the subject of performance-based testing would be helpful, too. I’d like participants to think of a learning objective in their own area of expertise and how they would like to design items that test more than just rote knowledge.
What do you hope your participants will take away?
Participants should be able to walk away with a better understanding of the challenges associated with solid development of test items that test more than memorization of knowledge.
You’ve attended several Users Conferences. What prompts you to return?
I think this is my 7th or 8th User’s Conference. Every time I attend I walk away with a better understanding of testing and how Questionmark technologies are being used in creative ways to enhance the testing experience. The networking opportunities and access to testing experts from around the world are invaluable to me.