A streamlined system for survey administration and reporting

Joan Phaup 2013 (3)Posted by Joan Phaup

It’s great to talk to customers who will be presenting case studies at the Questionmark 2014 Users Conference. They all bring to their presentations the lessons they’ve learned from experience.

Conference participants have always taken a keen interest in how to use surveys effectively, so I was quite interested to find out from Scott Bybee, a training manager from Verizon who will be talking at the conference about Leveraging Questionmark’s Survey Capabilities Within a Multi-system Model.

What will you be sharing during your conference presentation?

A lot of it will be about how our surveys, which are mostly Level 1 evaluations for training events and Level 3 self-assessments. I will tell how we can use one generic survey template for all the courses that are being evaluated. We do this by passing parameters from our LMS into the special fields in Questionmark. I’ll also talk about how we integrate data from our LMS with the survey data to create detailed reports in a custom reporting system we built: We have everything we need to get very specific demographic reporting out of the system.

Scott Bybee

Scott Bybee

How is this approach helping you?

This system integrates reporting for all level 1 and 3 surveys. This provides us a single solution for all of our training-related reporting needs. Prior to this, we had to collect data from multiple systems and manually tie it all together. Before, we had a lot of different surveys being used by the business. It became hard to match up results due to variances in questions. With this approach, everyone sees the same set of questions and the quality of the reporting is much higher.

The alternative would have been to collect demographic information using drop-down lists, which we’d have to constantly update and maintain. There’s also the issue of the participant possibly choosing the wrong options from the drop-downs. This way, we are passing everything along for them. They can’t make a mistake. Another advantage is that automatically including that information means it takes less time for them to complete the survey.

Do you have a key piece of advice about how to get truly useful data from surveys?

Make sure you are asking the right kinds of questions and are not trying to put too much into one question. Also, consider passing information directly from your LMS into Questionmark, so participants can’t make a mistake filling out a drop-down.

What do you hope people will take away from your session?

I hope they find out there are some really creative ways to use Questionmark to get what you want. For instance, we realized that by using Perception Integration Protocol (PIP), we could pass in all the variables needed for user-interface as well as alignment with back-end reporting.  I also want them to appreciate how much can be done by tying different systems together. The investment to make Questionmark work for surveys as well as assessments dramatically increased our return on investment (ROI).

What do you hope to take away from the conference?

This will be my fourth one to go to. Every time I go I learn something from the people who are there – things I’d never even thought about. I want to learn from people who are using the tool in innovative ways, and I also want to hear about where things are going in the future.

The conference agenda is taking shape here. You can save $200 if you register for the conference by December 12.


Conference Close-up: An Update on Flash and Captivate Simulations in Questions

Posted by Joan Phaup

There’s a lot to be learned at the Questionmark Users Conference from customers who have in-depth experience with online assessments. Doug Peterson from Verizon Communications brings firsthand knowledge of  developing Flash and Captivate questions  for use within high-stakes tests. His previous session on this subject was very well received, and he’s learned so much more in the past year that he will be revisiting in a presentation on Using Captivate and Flash Simulations in eLearning and Assessments.

I checked in with Doug the other day to ask him about his progress over the past year and to learn about his plans for the conference.


Doug Peterson

Q: Tell me what you have been doing at Verizon Communications lately.

A: I’m almost totally consumed with the automated tests for our call center agents’ training. These tests are of course delivered through Questionmark Perception and it has been a lot of work to keep up with these because the stuff that we’re testing is constantly changing. During the last year I have developed a new architectural approach for using Flash as a question type in Perception, and it’s worked out pretty well. I expect it to save me a lot of time I used to spend on maintenance.

Q:  Can you tell me more about how Questionmark fits into your testing program?

A: These are high-stakes tests. Agent trainees must pass three tests to keep their jobs.  They used to be written tests, but we found that having to store the results for several years meant that all the paperwork was becoming a little voluminous. By delivering through Questionmark, everything stored electronically. I can make  backups for every assessment, so that at any time someone  says ,”This person took this test on this date” and asks how they did, I can tell them exactly. By switching to Perception, e made grading the tests completely objective. We removed the temptation of instructors to sometimes subconsciously feel bad about failing someone and wanting to be more lenient in their grading. Over a six-week training course you kind of bond with students. You don’t want to hurt anybody but you also don’t want to send someone out on the call center floor if they’re not qualified. We took all that issue away by switching to online testing.

The ability to use Flash and Captivate in Questionmark  tests is crucial. There are things you can’t ask using pencil and paper! In Perception, when you build in an application simulation, the question is basically, “Use this application to accomplish a certain task.” This allows us to test agents’ knowledge much more thoroughly — to test things we just couldn’t test before. For example, we can look at their ability to fill out certain fields in their customer service forms with certain values. They have to show that they know which fields need to be filled in, and they truly know how to use the application.

Q: What do you expect people to learn during your presentation at this year’s users conference?

I’ll be sharing what I’ve been doing for the last couple of years so that they can get up to the speed and benefit from what I have learned.  I’ll give them tips, tricks and best practices in developing a simulation, deploying it to Perception, getting it to report the useful information they want to see and balancing that with maintenance issue. I will help them consider what outcomes they would like to appear in the reports, how specific they want to make those outcomes and how generic they want to keep them for maintenance reasons.. I’ll give them some files they can download and tear apart: examples of flash files and things like that.

Q: Have you gained some new insights in the past year?

A: Absolutely. The forms used by our agents are changed frequently — either the values in the menus change or new fields are added. I’m dealing with three modules and two tests per module.  Agents take a test after each module. If they fail they go through remediation and take the second test. It’s not a duplicate of the first test but it covers the same material.  I probably have 60 simulation questions within those tests.  Until recently, I would sometimes have to go in and make the same change in 60 different Flash files! So I actually broke the simulation out into two parts: the core modulation, which I call my “Grand Master Flash” — a generic simulation — and a question-specific Flash file that calls the Grand Master Flash and says, “Display this, make this field active,” and so on, and the Grand Master reads all these instructions. I basically build the simulation dynamically. So now if they change something about the form, 90% of the time I can go to my Grand Master Flash and make one change, one time and load that up to the master. There are still certain situations where I need to update individual questions, but since the question Flashes are much smaller files than I was working with before, it’s much easier to do this. So far, this new set-up is worth its weight in gold!

Q:   What keeps you coming back to the conference?

A: There is so much good stuff at these conferences that I know I’m going to pick up some new and important information every time I attend.  Really, the only bad thing that I have encountered in the conferences is that I’ve attended is that I can’t get to all of the seminars I want to go to!

This year I’m definitely going to Greg Pope’s presentation on Principles of Item and Test Analysis. I am in the middle of generating these reports and we’re preparing a new release of the test. I’m correcting some scoring issues and I’m  trying to analyze these tests and figure out which questions need help, to figure out if  we have a good balance. Greg has helped me a great deal in understanding the mean, the median, the mode, the standard deviation — what these things mean — and I want to get more of that when I go out to Los Angeles.

There’s so much good information, that quite frankly I would recommend to anyone who’s going, if their employer will allow it, send two people so that they can double up the amount of information they take back to work.

If your organization does send more than one person to the conference there are group discounts available for the conference, which will take place March 15 – 18 in Los Angeles. Whether you sign up individually or as a group, register by January 21st for early-bird savings.