Coming soon to Los Angeles: A full conference program by and for Questionmark users

Joan Phaup

Posted by Joan Phaup

Questionmark users who join us in Los Angeles for the Questionmark Users Conference  March 15 – 18 will be spoiled for choice when it comes to choosing breakout sessions. Many delegates will be presenting case studies and leading peer discussions. And we’ve put together product training, best practice and drop-in demo sessions to suit everyone from beginners to power users. 

Here’s the complete conference line-up. You can Click here for details about all these sessions:

General Sessions

  • Opening General Session: Would You “Like” To Know? The Roles of Assessment in Social Learning
  •  Keynote: Bryan Chapman on Assessment’s Strategic Role in Enterprise Learning: Innovation and Trends
  • Closing General Session: The Road Ahead

Case Studies

  • Tiered Assessment Model for Internal Certification (Intermediate/Advanced) – Accenture
  • Applying Diagnostic Assessment in a Virtual Corporate University (All Experience Levels) – PricewaterhouseCoopers
  • Dynamically Generating Certificates from an Assessment (Intermediate/Advanced) –  Beckman Coulter, Inc.
  • Questionmark in Medical Education: Planning and Implementing Blended Course Exams (Beginning/Intermediate) –  Loma Linda University School of Medicine
  • Enabling Self-Service Reporting (Intermediate) – Accenture
  • Implementing E-Testing in the US Coast Guard: Challenges and Solutions (All Experience Levels)
  • Ensuring the Security and Integrity of Certification Tests (All Experience Levels) – Philadelphia Parking Authority
  • Using Captivate and Flash Simulations in eLearning and Assessments (Beginning/Intermediate) – Verizon Communications

Tech Training

  • Introduction to Questionmark Perception for Beginners
  • Advanced Authoring Techniques Using Authoring Manager (Intermediate/Advanced)
  • Planning Your Migration from Perception v4 to v5 (Intermediate/Advanced)
  • Configuring the User Experience and Understanding Templates in Perception v5 (Intermediate/Advanced)
  • Authoring with Questionmark Live – A hands-on introduction (Bring your own laptop!) (Beginning/Intermediate)
  • Analyzing and Sharing Assessment Results (Beginning/Intermediate)
  • Integrating Perception with Other Systems (Advanced)

Best Practices

  • Best Practices for Surveys and Course Evaluations (All Experience Levels) – Greg Pope, Analytics and Psychometrics Manager, Questionmark
  • Using Questionmark Perception to Make SharePoint an Effective Learning Platform (All Experience Levels) – John Kleeman, Chairman, Questionmark
  • Getting the Most from a Small Screen: Design Strategies for Mobile Devices (All Experience Levels) –  Silke Fleischer, Co-founder and CEO, ATIV Software
  • Principles of Item and Test Analysis (All Experience Levels) – Greg Pope, Analytics and Psychometrics Manager, Questionmark
  • Using the Latest Learning Research to Improve Your Questionmark Assessments (All Experience Levels) – John Kleeman, Chairman, Questionmark
  • Writing High-quality Assessments that Yield Meaningful Results (All Experience Levels) – Howard Eisenberg, Training and Consultancy Manager, Questionmark

Peer Discussions

  • Evaluating the Effectiveness of Your Training Programs: Metrics and ROI – Farmers Insurance
  • Using Printing and Scanning to Capture Survey and Test Results – PG&E Academy
  • Delivering Assessments at Academic Institutions: An Open Conversation- Rio Salado College
  • Improving Your Workflow: Ensuring smooth sailing from question creation to release  –  U.S. Coast Guard

Drop-in Demos

  • Using Assessments with Social/Informal Learning Tools
  • Perception v5 and Questionmark Live
  • Tools and Features for Enhancing Your Assessments and Reports

We’ll also be hosting drop-in sessions with Questionmark technicians and group discussions with our product owners about potential new offerings. You can register for the conference online or email for more information.

Questionmark’s first Assessment Innovation Award goes to…

Posted by Joan Phaup

Having admired for many years the creativity and can-do spirit of our customers, we’re pleased to announce the first-ever Questionmark Assessment Innovation Award, which we’ll present at the  Questionmark Users Conference in Los Angeles next month.

For this year’s honor, we’ve chosen Accenture for their work with more than 30 internal certification programs that combine training, experience  and (of course!) assessments – mostly medium- to high- stakes online tests as well as work product evaluations and interviews.

Allison Horn

Over the past three years, Accenture has moved ahead on many fronts to grow their internal assessment development and delivery capabilities to meet the increased demand for knowledge and skill assessments while upholding high quality standards. Moving to a three-tier assessment model, the company  has modified their test development processes so that the amount of time, money and effort invested in each test is determined by its overall stakes. Foundation level programs, for instance, now require significantly less  subject matter expert time, beta testing and item analysis than master level programs. In addition to more efficient development processes, the company is using new tools to automate item and test analysis more than ever before.

On the delivery side, Accenture has begun using testing kiosks for smaller locations and adopting standard models for reporting results. They’ve also enhanced the use of their Questionmark/LMS integration to give certification candidates a more consistent, familiar end-user experience.

Allison Horn,  Senior Manager – Global HR  at Accenture, will accept the award on Accenture’s behalf  in addition to co-presenting a case study on Enabling Self-Service Reporting. Two of her colleagues will be talking at the conference about Accenture’s Tiered Assessment Model for Internal Certification.

You can see the entire list of case study presentations, peer discussions, demos, best practice presentations, product training breakouts and general sessions, by clicking here. And you can register online now!

Measuring the Impact of Social Learning

julie-smallPosted by Julie Delazyn

“The Social Network” – a movie about Facebook founder Mark Zuckerberg — was a big winner at the recent Golden Globe Awards in Los Angeles. And we’re pleased to say that social networks will be high on the agenda as Questionmark users gather in LA (with black ties nowhere in sight!) March 15 – 18 for the Questionmark 2011 Users Conference.

We’ll be paying considerable attention there to the growing importance of social networks as one aspect of a major shift in how people learn. Many organizations, recognizing that the vast majority of  what we learn comes from firsthand experience and from other people rather than from formal learning and study, are turning to informal and ‘social’ learning initiatives that use portals, blogs, wikis, forums and other tools. At this year’s conference we’ll be highlighting many new features and applications of Questionmark’s Open Assessment Platform that work in concert with many commonly used informal/social learning technologies – from embedded and observational assessment to new user interfaces and tools for content evaluation. (See a sneak peek below — and come to the conference for more on this!)

We’re looking forward to seeing you at the conference in March – we’re making sure the conference agenda will be packed with sessions and events that we’re sure you’ll be able to “like.”

Only two more days to register before the early-bird deadline!

Conference Close-up: An Update on Flash and Captivate Simulations in Questions

Posted by Joan Phaup

There’s a lot to be learned at the Questionmark Users Conference from customers who have in-depth experience with online assessments. Doug Peterson from Verizon Communications brings firsthand knowledge of  developing Flash and Captivate questions  for use within high-stakes tests. His previous session on this subject was very well received, and he’s learned so much more in the past year that he will be revisiting in a presentation on Using Captivate and Flash Simulations in eLearning and Assessments.

I checked in with Doug the other day to ask him about his progress over the past year and to learn about his plans for the conference.


Doug Peterson

Q: Tell me what you have been doing at Verizon Communications lately.

A: I’m almost totally consumed with the automated tests for our call center agents’ training. These tests are of course delivered through Questionmark Perception and it has been a lot of work to keep up with these because the stuff that we’re testing is constantly changing. During the last year I have developed a new architectural approach for using Flash as a question type in Perception, and it’s worked out pretty well. I expect it to save me a lot of time I used to spend on maintenance.

Q:  Can you tell me more about how Questionmark fits into your testing program?

A: These are high-stakes tests. Agent trainees must pass three tests to keep their jobs.  They used to be written tests, but we found that having to store the results for several years meant that all the paperwork was becoming a little voluminous. By delivering through Questionmark, everything stored electronically. I can make  backups for every assessment, so that at any time someone  says ,”This person took this test on this date” and asks how they did, I can tell them exactly. By switching to Perception, e made grading the tests completely objective. We removed the temptation of instructors to sometimes subconsciously feel bad about failing someone and wanting to be more lenient in their grading. Over a six-week training course you kind of bond with students. You don’t want to hurt anybody but you also don’t want to send someone out on the call center floor if they’re not qualified. We took all that issue away by switching to online testing.

The ability to use Flash and Captivate in Questionmark  tests is crucial. There are things you can’t ask using pencil and paper! In Perception, when you build in an application simulation, the question is basically, “Use this application to accomplish a certain task.” This allows us to test agents’ knowledge much more thoroughly — to test things we just couldn’t test before. For example, we can look at their ability to fill out certain fields in their customer service forms with certain values. They have to show that they know which fields need to be filled in, and they truly know how to use the application.

Q: What do you expect people to learn during your presentation at this year’s users conference?

I’ll be sharing what I’ve been doing for the last couple of years so that they can get up to the speed and benefit from what I have learned.  I’ll give them tips, tricks and best practices in developing a simulation, deploying it to Perception, getting it to report the useful information they want to see and balancing that with maintenance issue. I will help them consider what outcomes they would like to appear in the reports, how specific they want to make those outcomes and how generic they want to keep them for maintenance reasons.. I’ll give them some files they can download and tear apart: examples of flash files and things like that.

Q: Have you gained some new insights in the past year?

A: Absolutely. The forms used by our agents are changed frequently — either the values in the menus change or new fields are added. I’m dealing with three modules and two tests per module.  Agents take a test after each module. If they fail they go through remediation and take the second test. It’s not a duplicate of the first test but it covers the same material.  I probably have 60 simulation questions within those tests.  Until recently, I would sometimes have to go in and make the same change in 60 different Flash files! So I actually broke the simulation out into two parts: the core modulation, which I call my “Grand Master Flash” — a generic simulation — and a question-specific Flash file that calls the Grand Master Flash and says, “Display this, make this field active,” and so on, and the Grand Master reads all these instructions. I basically build the simulation dynamically. So now if they change something about the form, 90% of the time I can go to my Grand Master Flash and make one change, one time and load that up to the master. There are still certain situations where I need to update individual questions, but since the question Flashes are much smaller files than I was working with before, it’s much easier to do this. So far, this new set-up is worth its weight in gold!

Q:   What keeps you coming back to the conference?

A: There is so much good stuff at these conferences that I know I’m going to pick up some new and important information every time I attend.  Really, the only bad thing that I have encountered in the conferences is that I’ve attended is that I can’t get to all of the seminars I want to go to!

This year I’m definitely going to Greg Pope’s presentation on Principles of Item and Test Analysis. I am in the middle of generating these reports and we’re preparing a new release of the test. I’m correcting some scoring issues and I’m  trying to analyze these tests and figure out which questions need help, to figure out if  we have a good balance. Greg has helped me a great deal in understanding the mean, the median, the mode, the standard deviation — what these things mean — and I want to get more of that when I go out to Los Angeles.

There’s so much good information, that quite frankly I would recommend to anyone who’s going, if their employer will allow it, send two people so that they can double up the amount of information they take back to work.

If your organization does send more than one person to the conference there are group discounts available for the conference, which will take place March 15 – 18 in Los Angeles. Whether you sign up individually or as a group, register by January 21st for early-bird savings.

Registration for the Questionmark 2011 Users Conference now open

julie-smallPosted by Julie Delazyn

It’s that time of the year again!  Registration for the 2011 Users Conference in Los Angeles March 15 – 18  is now open.

We’re very pleased that Bryan Chapman will present the conference keynote address. As Chief Learning Strategist at Chapman Alliance, Bryan is involved in providing research-centric consulting solutions that help organizations define, operate and optimize their strategic learning initiatives.

Reasons to attend:

• Get technical training and tips on best practices
• Network with colleagues who share your interests
• Learn from your peers through case studies and
facilitated discussions
• Influence future Questionmark technologies and services
• Develop new skills and strategies
• Enjoy the sights and sounds of Los Angeles!

The conference is a valuable opportunity to network with other assessment professionals and to help shape future product features by meeting with Questionmark’s product managers. Be a part of our industry-specific discussions, birds-of-a-feather conversations, and evening events.

Want to learn more? Watch Questionmark CEO Eric Shepherd’s video:

What: 2011 Users Conference in Los Angeles
When: Tuesday, March 15th through Friday March 18th, 2011
Where: The Omni Los Angeles Hotel / Los Angeles, California

Reminder: group discounts available.  Also, register by December 3rd to save $200.

What’s new in online & mobile assessment? Find out at a Breakfast Briefing

Joan Phaup

Posted by Joan Phaup

Our first U.S. Questionmark Breakfast Briefings for 2010 are set for Boston and New York next week, so it seems like a good time to remind you about our annual visits to various cities. Questionmark staff look forward to greeting you at a briefing and sharing the latest news about online and mobile assessment.

Each briefing will include a complimentary breakfast at 8 a.m., followed by presentations and discussions from 8:30to about 11:30 a.m. There will be an overview of Questionmark Perception and a demonstration of Questionmark Live browser-based authoring, which makes it easy for subject matter experts to write everything from individual questions to entire assessments, including course evaluations.

We’ll cover lots of other topics  too, including:

* Delivering a single assessment to multiple devices, with centralized processing of results
* Using mobile devices such as Android phones and the Apple iPhone and iPad for quick course evaluations, observational assessments and more
* Assessments embedded within learning mashups on wikis, blogs and portals
* Integrations of Questionmark with SAP, SharePoint and other systems
* New tools for analyzing the results of surveys, quizzes and tests
* More meaningful reporting on the results of course evaluations

There will be a Questionmark User Group lunch and meeting after each of these briefings for people who are already using Questionmark Perception. User Groups offer an excellent opportunity to talk in depth about new technologies, to exchange ideas and to ask questions and get answers.

Here’s the schedule:

* Boston, MA (Woburn) –  Tuesday, September 21
* New York, NY   – Thursday, September 23
* Chicago, IL  – Monday, September 27
* Dallas, TX  – Tuesday, October 19
* Washington, DC  (Bethesda) – Thursday, October 21
* Ft Lauderdale, FL – Monday,October 26
* Los Angeles, CA  (Agoura Hills) – Tuesday, November 9

You can find complete details and sign up for a Breakfast Briefing and/or User Group meeting by visiting