Simulating real life: Questions that test application of knowledge

Doug Peterson HeadshotPosted By Doug Peterson

Questionmark offers over 20 different question formats. We have the standard multiple choice, multiple response, true/false and yes/no question types. We also have question types with more interaction, to more fully engage the participant: hot spot and drag-and-drop, fill in blanks and select a blank, and matching and ordering. We also have text match and essay questions that allow participants to show their understanding by formulating a response from scratch.

Questionmark has two other question types that I’d like to discuss, because I feel they are very powerful yet often overlooked: the Adobe Captivate and Flash question types.

Prior to joining Questionmark, I worked for a large telecommunications company, and one of my group’s responsibilities was to train customer call center employees. The call center representatives had to know all about hooking up set-top boxes and DVD players as well as configuring routers and setting up wireless connections. They had to know how to use several different trouble-shooting applications as well as the main trouble ticket application. We used Captivate and Flash question types in a few different ways to effectively and accurately assess the participants’ knowledge as they went through their 13 weeks of training.

  1.  We used Flash and ActionScript to create a duplicate of the trouble ticket application. And I mean *duplicate*. It looked and behaved exactly like the real thing. Honestly, there were a couple of times I got confused as to which was the real thing and which was the simulation when I had them open side by side, that’s how realistic the simulation was. With this simulation, we were able to go beyond using multiple choice questions that just asked, “What value would you select for Trouble Reason?” or “What error code would you use, given the customer’s description?” Instead, we presented the participant with (what appeared to be and behaved exactly like) the trouble ticket application and said, “Fill out the ticket.” We gave a point or two for every value they entered correctly, or checkbox they checked correctly, or radio button they selected correctly. In this way we could assess their understanding of what fields needed to be populated at every stage of the process and their overall ability to use the software, as well as their understanding of what values to use in each field.
  2.  Similar to #1, we created simulations for setting up a wireless connection or configuring a router. We presented the participant with a Windows desktop and they had to go through the process of setting up a connection to a local router – entering the SSID, entering the WEP key, etc. We didn’t give points for individual steps in this one, as the instructions were to set up a connection – either you did it all correctly and established the connection, or you
  3.  The class was typically taught in a classroom, and at one point the instructor would wheel in an audio/visual cart with a television, a set-top box, a DVD player, and a home theater sound system. The members of the class would then wire the components together correctly, or troubleshoot the instructor’s incorrect wiring job. Then one day we were asked to teach the class remotely, with student’s taking the training in their own homes. How could we do the wiring exercise if the students weren’t all in the same physical location? As I’m sure you’ve guessed, we used a Flash simulation. The simulation presented the backs of the various components along with the ability to select different types of wiring (HDMI cable, coax cable, and RCA cables). Students could click-and-drag the selected wire from a connector on one component to a connector on another component.

Because this way of assessing a learner’s understanding is not all that common, we used similar simulations as formative quizzes during the training and provided a practice assessment prior to the first “real” assessment. This helped the participants get comfortable with the format by the time it really counted, which is important: We want to be fair to the learner and make sure we give them every opportunity to prove their knowledge, skill or ability without interference or stress. It’s not fair to suddenly spring a new question format on them that they’ve never seen before.

One great way to learn more about this topic is to attend our the Questionmark 2014 Users Conference March 4-7 in San Antonio. I typically present a session on using Flash and Captivate in e-learning and assessments. Looking forward to seeing you there!

Conference Close-up: An Update on Flash and Captivate Simulations in Questions

Posted by Joan Phaup

There’s a lot to be learned at the Questionmark Users Conference from customers who have in-depth experience with online assessments. Doug Peterson from Verizon Communications brings firsthand knowledge of  developing Flash and Captivate questions  for use within high-stakes tests. His previous session on this subject was very well received, and he’s learned so much more in the past year that he will be revisiting in a presentation on Using Captivate and Flash Simulations in eLearning and Assessments.

I checked in with Doug the other day to ask him about his progress over the past year and to learn about his plans for the conference.


Doug Peterson

Q: Tell me what you have been doing at Verizon Communications lately.

A: I’m almost totally consumed with the automated tests for our call center agents’ training. These tests are of course delivered through Questionmark Perception and it has been a lot of work to keep up with these because the stuff that we’re testing is constantly changing. During the last year I have developed a new architectural approach for using Flash as a question type in Perception, and it’s worked out pretty well. I expect it to save me a lot of time I used to spend on maintenance.

Q:  Can you tell me more about how Questionmark fits into your testing program?

A: These are high-stakes tests. Agent trainees must pass three tests to keep their jobs.  They used to be written tests, but we found that having to store the results for several years meant that all the paperwork was becoming a little voluminous. By delivering through Questionmark, everything stored electronically. I can make  backups for every assessment, so that at any time someone  says ,”This person took this test on this date” and asks how they did, I can tell them exactly. By switching to Perception, e made grading the tests completely objective. We removed the temptation of instructors to sometimes subconsciously feel bad about failing someone and wanting to be more lenient in their grading. Over a six-week training course you kind of bond with students. You don’t want to hurt anybody but you also don’t want to send someone out on the call center floor if they’re not qualified. We took all that issue away by switching to online testing.

The ability to use Flash and Captivate in Questionmark  tests is crucial. There are things you can’t ask using pencil and paper! In Perception, when you build in an application simulation, the question is basically, “Use this application to accomplish a certain task.” This allows us to test agents’ knowledge much more thoroughly — to test things we just couldn’t test before. For example, we can look at their ability to fill out certain fields in their customer service forms with certain values. They have to show that they know which fields need to be filled in, and they truly know how to use the application.

Q: What do you expect people to learn during your presentation at this year’s users conference?

I’ll be sharing what I’ve been doing for the last couple of years so that they can get up to the speed and benefit from what I have learned.  I’ll give them tips, tricks and best practices in developing a simulation, deploying it to Perception, getting it to report the useful information they want to see and balancing that with maintenance issue. I will help them consider what outcomes they would like to appear in the reports, how specific they want to make those outcomes and how generic they want to keep them for maintenance reasons.. I’ll give them some files they can download and tear apart: examples of flash files and things like that.

Q: Have you gained some new insights in the past year?

A: Absolutely. The forms used by our agents are changed frequently — either the values in the menus change or new fields are added. I’m dealing with three modules and two tests per module.  Agents take a test after each module. If they fail they go through remediation and take the second test. It’s not a duplicate of the first test but it covers the same material.  I probably have 60 simulation questions within those tests.  Until recently, I would sometimes have to go in and make the same change in 60 different Flash files! So I actually broke the simulation out into two parts: the core modulation, which I call my “Grand Master Flash” — a generic simulation — and a question-specific Flash file that calls the Grand Master Flash and says, “Display this, make this field active,” and so on, and the Grand Master reads all these instructions. I basically build the simulation dynamically. So now if they change something about the form, 90% of the time I can go to my Grand Master Flash and make one change, one time and load that up to the master. There are still certain situations where I need to update individual questions, but since the question Flashes are much smaller files than I was working with before, it’s much easier to do this. So far, this new set-up is worth its weight in gold!

Q:   What keeps you coming back to the conference?

A: There is so much good stuff at these conferences that I know I’m going to pick up some new and important information every time I attend.  Really, the only bad thing that I have encountered in the conferences is that I’ve attended is that I can’t get to all of the seminars I want to go to!

This year I’m definitely going to Greg Pope’s presentation on Principles of Item and Test Analysis. I am in the middle of generating these reports and we’re preparing a new release of the test. I’m correcting some scoring issues and I’m  trying to analyze these tests and figure out which questions need help, to figure out if  we have a good balance. Greg has helped me a great deal in understanding the mean, the median, the mode, the standard deviation — what these things mean — and I want to get more of that when I go out to Los Angeles.

There’s so much good information, that quite frankly I would recommend to anyone who’s going, if their employer will allow it, send two people so that they can double up the amount of information they take back to work.

If your organization does send more than one person to the conference there are group discounts available for the conference, which will take place March 15 – 18 in Los Angeles. Whether you sign up individually or as a group, register by January 21st for early-bird savings.