Posted by Joan Phaup
There’s a lot to be learned at the Questionmark Users Conference from customers who have in-depth experience with online assessments. Doug Peterson from Verizon Communications brings firsthand knowledge of developing Flash and Captivate questions for use within high-stakes tests. His previous session on this subject was very well received, and he’s learned so much more in the past year that he will be revisiting in a presentation on Using Captivate and Flash Simulations in eLearning and Assessments.
I checked in with Doug the other day to ask him about his progress over the past year and to learn about his plans for the conference.
Q: Tell me what you have been doing at Verizon Communications lately.
A: I’m almost totally consumed with the automated tests for our call center agents’ training. These tests are of course delivered through Questionmark Perception and it has been a lot of work to keep up with these because the stuff that we’re testing is constantly changing. During the last year I have developed a new architectural approach for using Flash as a question type in Perception, and it’s worked out pretty well. I expect it to save me a lot of time I used to spend on maintenance.
Q: Can you tell me more about how Questionmark fits into your testing program?
A: These are high-stakes tests. Agent trainees must pass three tests to keep their jobs. They used to be written tests, but we found that having to store the results for several years meant that all the paperwork was becoming a little voluminous. By delivering through Questionmark, everything stored electronically. I can make backups for every assessment, so that at any time someone says ,”This person took this test on this date” and asks how they did, I can tell them exactly. By switching to Perception, e made grading the tests completely objective. We removed the temptation of instructors to sometimes subconsciously feel bad about failing someone and wanting to be more lenient in their grading. Over a six-week training course you kind of bond with students. You don’t want to hurt anybody but you also don’t want to send someone out on the call center floor if they’re not qualified. We took all that issue away by switching to online testing.
The ability to use Flash and Captivate in Questionmark tests is crucial. There are things you can’t ask using pencil and paper! In Perception, when you build in an application simulation, the question is basically, “Use this application to accomplish a certain task.” This allows us to test agents’ knowledge much more thoroughly — to test things we just couldn’t test before. For example, we can look at their ability to fill out certain fields in their customer service forms with certain values. They have to show that they know which fields need to be filled in, and they truly know how to use the application.
Q: What do you expect people to learn during your presentation at this year’s users conference?
I’ll be sharing what I’ve been doing for the last couple of years so that they can get up to the speed and benefit from what I have learned. I’ll give them tips, tricks and best practices in developing a simulation, deploying it to Perception, getting it to report the useful information they want to see and balancing that with maintenance issue. I will help them consider what outcomes they would like to appear in the reports, how specific they want to make those outcomes and how generic they want to keep them for maintenance reasons.. I’ll give them some files they can download and tear apart: examples of flash files and things like that.
Q: Have you gained some new insights in the past year?
A: Absolutely. The forms used by our agents are changed frequently — either the values in the menus change or new fields are added. I’m dealing with three modules and two tests per module. Agents take a test after each module. If they fail they go through remediation and take the second test. It’s not a duplicate of the first test but it covers the same material. I probably have 60 simulation questions within those tests. Until recently, I would sometimes have to go in and make the same change in 60 different Flash files! So I actually broke the simulation out into two parts: the core modulation, which I call my “Grand Master Flash” — a generic simulation — and a question-specific Flash file that calls the Grand Master Flash and says, “Display this, make this field active,” and so on, and the Grand Master reads all these instructions. I basically build the simulation dynamically. So now if they change something about the form, 90% of the time I can go to my Grand Master Flash and make one change, one time and load that up to the master. There are still certain situations where I need to update individual questions, but since the question Flashes are much smaller files than I was working with before, it’s much easier to do this. So far, this new set-up is worth its weight in gold!
Q: What keeps you coming back to the conference?
A: There is so much good stuff at these conferences that I know I’m going to pick up some new and important information every time I attend. Really, the only bad thing that I have encountered in the conferences is that I’ve attended is that I can’t get to all of the seminars I want to go to!
This year I’m definitely going to Greg Pope’s presentation on Principles of Item and Test Analysis. I am in the middle of generating these reports and we’re preparing a new release of the test. I’m correcting some scoring issues and I’m trying to analyze these tests and figure out which questions need help, to figure out if we have a good balance. Greg has helped me a great deal in understanding the mean, the median, the mode, the standard deviation — what these things mean — and I want to get more of that when I go out to Los Angeles.
There’s so much good information, that quite frankly I would recommend to anyone who’s going, if their employer will allow it, send two people so that they can double up the amount of information they take back to work.
If your organization does send more than one person to the conference there are group discounts available for the conference, which will take place March 15 – 18 in Los Angeles. Whether you sign up individually or as a group, register by January 21st for early-bird savings.