Better outcomes make the outcome better!
Posted by Joan Phaup
As we build the program for the Questionmark 2014 Users Conference, I’m having a great time chatting with presenters about their plans.
I spoke recently with Gail Watson from the US Marine Corps University’s College of Distance Education and Training. As an institution that educates large numbers of people about large and complex subjects, the college has grappled with how to make sure the tests it administers yield meaningful outcomes.
Gail says that careful attention to the relationship of business practices and organization of the folder/topic system is critical to the effective delivery of a Questionmark assessment if the subject matter is broad or complex. When subject matter experts (SMEs) are made aware of Questionmark’s capabilities prior to creating questions, they can organize topics and questions in ways that result in better feedback. The success of the assessments and topics can be better measured.
Her case study presentation in San Antonio will also focus on principles for using multiple response, pull-down, matching and ranking questions.
Who would benefit most from your presentation?
I think it will help users who either have broad, complex material to be assessed as well as new users who may be confused about where to start. Even if you are knowledgeable about Questionmark, sometimes a subject is so big you don’t know how to set up your topics or use Question blocks — things like that.
How will you help your audience?
I’d like to show how to assess the capabilities of Questionmark in order to make assessments fit in with (business) processes and decision points. It’s a matter of matching up the SME vocabulary with the Questionmark vocabulary. We have done this to such an extent that we now have a roadmap that we go through long before any items are put into Questionmark. I can look at a question number and tell from it exactly what topic it relates to and what folders it goes in.
I’d like to show the importance of understanding Questionmark’s capabilities and how they relate to the world of SMEs – so that people understand how to match their assessment up with decision points up front. You can’t do that after the fact. By getting this right, you end up providing good feedback to students, good feedback to SMEs and good feedback to managers.
Do your SMEs use Questionmark Live to author items?
Yes, and we use it heavily in our question review and approval process. This actually relates to the importance of the question number used in the Description Field. I work with large batches of approved questions, and without that question number, I don’t know what folder “path” to put them in. An example question number is 5212.4.1. I can tell just from looking at number that this question resides in a topic folder 4 levels down. So as I start receiving questions, I can start producing assessments.
How do question and topic outcomes influence assessment results?
Creating good question outcomes for multiple select, matching, pull down and ranking questions produces supporting data for item analysis of those question types. These types of questions can tell you what correct answers people are missing, what wrong answers are being collected, and so forth. Creating good topic outcomes give Subject Matter Experts (SME’s) insight into weak areas of knowledge from topic outcome reports and gives failing students the proper assessment feedback to focus their study for re-test
Can you share some quick tips for organizing topics and folders effectively?
List your SME “vocabulary” and processes, then list Questionmark’s “vocabulary” and capabilities. After that, match them up to create a topic folder and assessment map; a question management process; an assessment management process; and a review and maintenance process.
What do you hope people will take away from your session?
Everyone who touches the assessment process needs to know how Questionmark works so that they can best leverage Questionmark’s capabilities. I’d like to help people match up what Questionmark can do with what they need to accomplish in their learning organizations. I also will be talking about Questionmark Live, because we use it quite heavily in the process of creating questions. I hope they’ll learn some tips for using it effectively.
Early-bird registration for the conference, to be held in San Antonio March 4 – 7, is open through December 12. Click here to register.