Conference Close-up: An Update on Flash and Captivate Simulations in Questions

Posted by Joan Phaup

There’s a lot to be learned at the Questionmark Users Conference from customers who have in-depth experience with online assessments. Doug Peterson from Verizon Communications brings firsthand knowledge of  developing Flash and Captivate questions  for use within high-stakes tests. His previous session on this subject was very well received, and he’s learned so much more in the past year that he will be revisiting in a presentation on Using Captivate and Flash Simulations in eLearning and Assessments.

I checked in with Doug the other day to ask him about his progress over the past year and to learn about his plans for the conference.


Doug Peterson

Q: Tell me what you have been doing at Verizon Communications lately.

A: I’m almost totally consumed with the automated tests for our call center agents’ training. These tests are of course delivered through Questionmark Perception and it has been a lot of work to keep up with these because the stuff that we’re testing is constantly changing. During the last year I have developed a new architectural approach for using Flash as a question type in Perception, and it’s worked out pretty well. I expect it to save me a lot of time I used to spend on maintenance.

Q:  Can you tell me more about how Questionmark fits into your testing program?

A: These are high-stakes tests. Agent trainees must pass three tests to keep their jobs.  They used to be written tests, but we found that having to store the results for several years meant that all the paperwork was becoming a little voluminous. By delivering through Questionmark, everything stored electronically. I can make  backups for every assessment, so that at any time someone  says ,”This person took this test on this date” and asks how they did, I can tell them exactly. By switching to Perception, e made grading the tests completely objective. We removed the temptation of instructors to sometimes subconsciously feel bad about failing someone and wanting to be more lenient in their grading. Over a six-week training course you kind of bond with students. You don’t want to hurt anybody but you also don’t want to send someone out on the call center floor if they’re not qualified. We took all that issue away by switching to online testing.

The ability to use Flash and Captivate in Questionmark  tests is crucial. There are things you can’t ask using pencil and paper! In Perception, when you build in an application simulation, the question is basically, “Use this application to accomplish a certain task.” This allows us to test agents’ knowledge much more thoroughly — to test things we just couldn’t test before. For example, we can look at their ability to fill out certain fields in their customer service forms with certain values. They have to show that they know which fields need to be filled in, and they truly know how to use the application.

Q: What do you expect people to learn during your presentation at this year’s users conference?

I’ll be sharing what I’ve been doing for the last couple of years so that they can get up to the speed and benefit from what I have learned.  I’ll give them tips, tricks and best practices in developing a simulation, deploying it to Perception, getting it to report the useful information they want to see and balancing that with maintenance issue. I will help them consider what outcomes they would like to appear in the reports, how specific they want to make those outcomes and how generic they want to keep them for maintenance reasons.. I’ll give them some files they can download and tear apart: examples of flash files and things like that.

Q: Have you gained some new insights in the past year?

A: Absolutely. The forms used by our agents are changed frequently — either the values in the menus change or new fields are added. I’m dealing with three modules and two tests per module.  Agents take a test after each module. If they fail they go through remediation and take the second test. It’s not a duplicate of the first test but it covers the same material.  I probably have 60 simulation questions within those tests.  Until recently, I would sometimes have to go in and make the same change in 60 different Flash files! So I actually broke the simulation out into two parts: the core modulation, which I call my “Grand Master Flash” — a generic simulation — and a question-specific Flash file that calls the Grand Master Flash and says, “Display this, make this field active,” and so on, and the Grand Master reads all these instructions. I basically build the simulation dynamically. So now if they change something about the form, 90% of the time I can go to my Grand Master Flash and make one change, one time and load that up to the master. There are still certain situations where I need to update individual questions, but since the question Flashes are much smaller files than I was working with before, it’s much easier to do this. So far, this new set-up is worth its weight in gold!

Q:   What keeps you coming back to the conference?

A: There is so much good stuff at these conferences that I know I’m going to pick up some new and important information every time I attend.  Really, the only bad thing that I have encountered in the conferences is that I’ve attended is that I can’t get to all of the seminars I want to go to!

This year I’m definitely going to Greg Pope’s presentation on Principles of Item and Test Analysis. I am in the middle of generating these reports and we’re preparing a new release of the test. I’m correcting some scoring issues and I’m  trying to analyze these tests and figure out which questions need help, to figure out if  we have a good balance. Greg has helped me a great deal in understanding the mean, the median, the mode, the standard deviation — what these things mean — and I want to get more of that when I go out to Los Angeles.

There’s so much good information, that quite frankly I would recommend to anyone who’s going, if their employer will allow it, send two people so that they can double up the amount of information they take back to work.

If your organization does send more than one person to the conference there are group discounts available for the conference, which will take place March 15 – 18 in Los Angeles. Whether you sign up individually or as a group, register by January 21st for early-bird savings.

Conference Close-up: Using Flash and Captivate Questions with Questionmark

Posted by Joan Phaup

Participants in the annual Questionmark Users Conference bring a lot of enthusiasm about using innovative question types in their assessments. A number of our customers have extedougnsive experience with this and like to share their expertise at the conference. I spoke the other day with Doug Peterson from Verizon Communications and asked him about the case study he will share at the conference about using Flash and Captivate questions  within Questionmark Perception.

Here’s a quick wrap-up of our conversation:

Q:  What’s your role at Verizon  Communications?
A:  I have two roles: I develop, maintain and deliver training — mainly  now on internet technologies — and I’m responsible for a series of online automated tests for our help center training program. This is a pass/fail curriculum and very high stakes because these tests can affect people’s job status. So we need to be absolutely sure that the tests are well written and well maintained. These used to be written tests that were graded by an instructor. We turned to Questionmark for an objective, unbiased, online, airtight testing system and I oversee that.

Q: How are you using Questionmark Perception?
A: We have a couple of tests for each of the three modules in the training curriculum. We use Questionmark for end-of-lesson reviews as well as the higher stakes tests  that determine whether a person has passed or failed a module. We use scenarios that trainees might encounter in working with a customer. There might be 6 to 8 scenarios in each test and 10 or 12 questions about each scenario. The trainees take these tests right in the classroom, on their classroom computers.  We create individual QM accounts for each student and schedule the tests directly for those accounts.  We schedule them for a specific day and time window.  No one can see the tests except for the students, and they can only access them during the testing window. We had subject matter experts tell us what we needed to cover in the scenarios and what questions we needed to ask about them. They explained what would be a reasonable way to present a question or simulation to test a particular skill. Once we’d created all the scenarios and written all the questions we did an in-depth validation.

Q: What will you be sharing during your case study presentation at the Users Conference?
A:  Our call center agents have to use several applications when they get a call from a customer. They’ll have to look up a trouble ticket, get information about the customer and so forth. We need to make sure they knew how to use those applications, so we have created Perception questions using Captivate and Flash files with ActionScript that present the application to the student. Then the student needs to work through the application to demonstrate their proficiency with it. We’ve worked out a way to create a highly interactive, very realistic simulation in Flash that captures each student’s actions in using a particular application. It really tracks step by step. Being able to take the individual things from the Flash scenarios makes it so that when we run reports after the test we can easily see if a lot of people are is missing something like clicking on a particular button. Then the instructor can go back and make sure the students understand what they are supposed to. We went through a complex process to figure all this out, but it’s given us the ability to create a highly interactive, very realistic simulation in Flash with action script ActionScript coding and all kinds of logic and still pass back individual point values for different tasks. I’m very proud of the tests we have created and the work we have done. We have some fabulous questions in there that allows the students to show that they really understand applications and know how to do something from start to finish. We learned many tips and tricks along the way and I will be sharing those with the people at my session.

Q: What are you looking forward to at this year’s conference?
A: I really enjoyed the sessions on item analysis and test validity at the 2009 conference, and I am looking forward to learning even more about those subjects this year. And anything about new functionality in Perception Version 5 will be on my list too.

You can attend Doug’s presentations and many others at the conference in Miami March 14 – 17. Early-bird registration ends January 22nd, so sign up soon!