Heading home from San Antonio

Joan Phaup 2013 (3)Posted by Joan Phaup

Bryan Chapman (2)

Bryan Chapman

As we head back home from this week’s Questionmark Users Conference in San Antonio, it’s good to reflect on the connections people made with one another during discussions, focus groups, social events and a wide variety of presentations covering best practices, case studies and the features and functions of Questionmark technologies. Many thanks to all our presenters!

Bryan Chapman’s keynote on Transforming Open Data into Meaning and Action offered an expansive approach to a key theme of this year’s conference. Bryan described the tremendous power of OData while dispelling much of the mystery around it. He explained that OData can be exchanged in simple ways, such as using a URL or inserting a command line to create, read, update, and/or delete data items.

thurs night scottIt was interesting to see how focusing on key indicators that have the biggest impact can produce easy-to-understand visual representations of what is happening within an organization. Among the many dashboards Bryan shared was on that showed the amount of safety training in relation to incidence of on-the-job injuries.thurs night trio

No conference is complete without social events that nurture new friendships and cement long-established bonds. Yesterday ended with a visit to the Rio Cibolo Ranch outside the city, where we enjoyed a Texas-style meal, western music and all manner of ranch activities. Many of us got acquainted with some Texas Longhorn Cattle, and the bravest folks of all took some lassoing lessons (snagging a  mechanical calf, not a longhorn!).

Today’s breakouts and general session complete three intensive days of learning. Here’s wishing everyone a good journey home and continued connections in the year ahead.

Early-birds: Check out the conference program and register by tomorrow

Joan Phaup 2013 (3)Posted by Joan Phaup

You can still get a $100 early-bird registration discount for the Questionmark 2014 Users Conference if you register by tomorrow (Thursday, January 30th).

The program for March 4 – 7 includes a keynote by Learning Strategist Bryan Chapman on the power of open data, which will be a hot topic throughout this gathering at the Grand Hyatt San Antonio.

You can click here for program details, but here’s the line-up:

Optional Pre-Conference Workshopsbreakout 2013

  • ​Questionmark Boot Camp: Basic Training for Beginners  ​(full day)
  • ​Test Development Fundamentals  (half day)
  • ​The Art & Craft of Item Writing   ​(half day)

Case Studies

  • Using Questionmark to Conduct Performance Based Certifications —  SpaceTEC®
  • Better Outcomes Make the Outcome Better! — US Marine Corps University
  • ​Generating and Sending Custom Completion Certificates —  The Aurelius Group
  • ​Leveraging Questionmark’s Survey Capabilities with a Multi-system Model – Verizon
  • ​Importing Questions into Questionmark Live on a Tri-Military Service Training Campus Medical Education & Training Campus
  • How Can a Randomly Designed Test be Fair to All? US Coast Guard Training Center

Best Practices doug teaching 2013

  • ​Principles of Psychometrics and Measurement Design
  • 7 Reasons to Use Online Assessments for Compliance
  • ​Reporting and Analytics: Understanding Assessment Results
  • ​Making it Real: Building Simulations Into Your Quizzes and Tests
  • ​Practical Lessons from Psychology Research to Improve Your Assessments
  • Item Writing Techniques for Surveys, Quizzes and Tests

Questionmark Features & Functions

  • Introduction to Questionmark for Beginners
  • BYOL: Item and Topic Authoring
  • BYOL: Collaborative Assessment Authoring
  • Integrating with Questionmark’s Open Assessment Platform
  • Using Questionmark’s OData API for Analytics (BYOL)
  • Successfully Deploying Questionmark Perception
  • Customizing the Participant Interface


  • Testing What We Teach: How can we elevate our effectiveness without additional time or resources?
  • Testing is Changing: Practical and Secure Assessment in the 21st Century

Future Solutions Focus Groups

  • Open Assessment Platform and Analytics
  • Authoring and Delivery

Special Interest Group Meetings

  • Military/Defense US DOD and Homeland Security
  • Utilities/Energy Generation and Distribution
  • Higher Education
  • Universities

Drop-in Demos of new Questionmark features and capabilities

Tech Central: Drop-in meetings with Questionmark technicians


Register for the conference by tomorrow, Thursday, January 31st, to get the early-bird discount.



Transforming Open Data into Meaning and Action: Q&A with Bryan Chapman

Joan Phaup 2013 (3)Learning strategist Bryan Chapman’s keynote address on Transforming Open Data into Meaning and Action will be a highlight of the Questionmark 2014 Users Conference in San Antonio March 4 – 7.

With customers now using the Questionmark OData API to harvest meaning from their assessment results with greater freedom and flexibility, we are excited to hear about the broader implications of open data and the opportunities for learning organizations to make the most of it.

The Open Data Protocol, an industry standard for accessing data sources via the web, provides new opportunities reporting and analyzing assessment results. OData feeds can be consumed and analyzed by many leading business intelligence applications, provide new options for custom reports and dashboards..

I asked Bryan recently about what the advent of OData will mean, particularly with regard to learning and measurement:

The concept of open data is being talked about a lot these days. Why is it important?

Bryan Chapman (2)

Bryan Chapman

I’ve got to confess that I’m a total data junkie, so I get very excited about the endless possibilities of open data. Think of how much data is being collected on a daily basis all over the world from scientific discoveries, government data, opinion polls, and even learning.  Gartner recently said that companies are collecting 300% more data than they did 4 years ago.  It’s crazy. But consider what we can discover by selectively combining data in meaningful ways – something OData enables us to do.

Here’s a non-learning example of how powerful data can be: The CDC (Centers for Disease Control) collects information showing incidence of heart disease on a county-by-county basis. A group called Third World Congress on Positive Psychology developed a way to parse through Twitter feeds across two counties, analyzing the use of 40,000 words in over 80 million tweets.

By combining these separate data feeds, they discovered a pattern between having a positive attitude and having lower risk of heart attack. This is a rather simple example, but just think of what kinds of patterns we will find as open data takes hold.

What is the significance of OData for testing and assessment professionals?

If you want to discover patterns of behavior that make companies successful, what better variable to plug in than testing and assessment data?

Case in point: A while ago I worked with a large software company.  Independent, outside data suggested that customers felt that the company’s help desk support team often lacked knowledge in specific technical areas.  We went in, created an assessment (using Questionmark, by the way!) and created a gap analysis across 60 technical skills areas for all help desk support.

We turned the data into a gap analysis heat map with red, yellow and green indicators showing a range of levels from expertise down to lack of knowledge.  When we presented their senior management with the map, it was very clear to see several things…where training was needed…sometimes having the wrong person on the wrong team, and a whole lot more.

This was great as far as it went, but it was just a single snapshot in time…it wasn’t ongoing. I think back on that project and wonder how much more impressive it would have been if the data was continually measured, linked to a dashboard and frequently compared to the independent audit of customer responses.

OData makes this possible.

How can the use of open data impact learning and performance?

First, with open data, it’s relatively easy to flow the results of learning, testing and assessment right into the performance review process.  I’ve been watching the major talent management vendors who have tools to conduct annual performance reviews, do staff planning, succession planning and pay for performance; many of them are gradually adding OData feeds (both in and out of their systems). So creating that level of interoperability is already starting to shape up. The bigger win is the ability to link learning with organizational performance: Kirkpatrick Level 4!

Most of us feel very comfortable collecting Level 1 and Level 2. Do they like the training?  And are they learning, comparing pre and post test score? Some get how to do level 3 by sending out delayed questionnaires asking what skills are being used on the job; or through performance observation. But open data is really the enabler for linking learning and performance with key company metrics, income, productivity, retention, and lots of bottom line results — especially as other parts of the business make their data available through open channels.

How will your keynote address relate to the specific interests of Questionmark customers?

I’m not a technical guru when it comes to open data…not by a long shot. There will be others there who can tell you all about Questionmark’s OData capabilities. But I really think the hardest part of this is re-imagining how data can be creatively combined to paint the whole picture, or at least understanding what others might do with data that we make available through testing and assessment.

I’ll be sharing several examples of innovative approaches, but that’s just the tip of the iceberg.  If our organizations are really collecting 300% more data than 4 years ago, there are simply way too many data streams to combine.  So we need to start off by keeping things simple – to figure out which data streams can get us where we need to go.

If do my job well, we’ll all start dreaming of ways we can marry data together and apply meaning. And before long, we can expect to see some very creative dashboards linking learning data with actual business performance.

Learn more about the conference program, which includes two new pre-conference workshops: Test Development Fundamentals and The Art and Craft of Item Writing. Register for the conference by January 30 to save $100. Another current learning opportunity: 3-day Questionmark training in Las Vegas February 4 – 6.

Podcast: Bryan Chapman on Assessment’s Strategic Role in Learning


Posted by Joan Phaup

I am looking forward to the Questionmark 2011 Users Conference for many reasons. For one thing, it will take me back to my hometown of Los Angeles! But more importantly, the conference will bring Questionmark staff together with our customers for three great days of learning and networking.

Bryan Chapman

A highlight of the conference will by a keynote by Bryan Chapman on Assessment’s Strategic Role in Enterprise Learning: Innovations and Trends.

As Chief Learning Strategist at Chapman Alliance, Bryan organizations define, operate and optimize their strategic learning initiatives. His experience with many different types of organizations will inform his talk, which will include examples of innovative assessment strategies that have helped shorten training delivery times, motivate learners and improve performance.

I asked  Bryan for a sneak preview of what he’ll be talking about the conference. You can listen in on our conversation here. But before you do, I’d like to remind Questionmark users that this Friday, December 3rd, is the last day for early-bird registration savings. So after you’ve listened to this podcast, get ready to register soon!

Registration for the Questionmark 2011 Users Conference now open

julie-smallPosted by Julie Delazyn

It’s that time of the year again!  Registration for the 2011 Users Conference in Los Angeles March 15 – 18  is now open.

We’re very pleased that Bryan Chapman will present the conference keynote address. As Chief Learning Strategist at Chapman Alliance, Bryan is involved in providing research-centric consulting solutions that help organizations define, operate and optimize their strategic learning initiatives.

Reasons to attend:

• Get technical training and tips on best practices
• Network with colleagues who share your interests
• Learn from your peers through case studies and
facilitated discussions
• Influence future Questionmark technologies and services
• Develop new skills and strategies
• Enjoy the sights and sounds of Los Angeles!

The conference is a valuable opportunity to network with other assessment professionals and to help shape future product features by meeting with Questionmark’s product managers. Be a part of our industry-specific discussions, birds-of-a-feather conversations, and evening events.

Want to learn more? Watch Questionmark CEO Eric Shepherd’s video:

What: 2011 Users Conference in Los Angeles
When: Tuesday, March 15th through Friday March 18th, 2011
Where: The Omni Los Angeles Hotel / Los Angeles, California

Reminder: group discounts available.  Also, register by December 3rd to save $200.