Reflections on Barcelona: Great learning, great connections

Doug Peterson HeadshotPosted By Doug Peterson

I had the opportunity to attend and present at the Questionmark European Users Conference in Barcelona, Spain, November 10 – 12. Questionmark2013_DSC3268 If you’ve never had the chance to go to Barcelona, I highly recommend it! This is a *beautiful* city full of charming people, wonderful architecture, and GREAT food.

Traveling and seeing new sights and engaging in new adventures is always fun, but the true “stars of the show” at a users conference are, of course, the users.

It was wonderful to catch up with customers I first met two years ago in Brussels. I also had the opportunity to meet in person several people I had met over the last couple of years only through emails and conference calls, and it was great to put a face with a name. And of course, it was wonderful to make brand new friends whom I hope to see again next year!

One of the things I enjoy the most about Questionmark Users Conferences is how customers learn from each other. This happens in a more structured way during the many sessions presented by members of our user community, but I enjoy it even more when it happens more informally.

During the opening reception Sunday night I had the opportunity to talk with several customers, to hear why they were attending – what they wanted to get out of the conference – and then introduce them to another customer or a Questionmark employee who could help them meet their goals for the conference. Breakfast and lunch conversations were always interesting, and even during Monday night’s fantastic dinner at El Torre Dels Lleones, the conversations between different users facing various challenges continued (except, of course, when we were all watching the flamenco dancers perform!). These conversations are simply invaluable, not only because customers help each other find answers to their challenges, but because I as an employee gain insights and a depth of understanding as to what our customers are doing and the problems they are facing in ways that an email or a phone call can’t communicate.

Questionmark2013_DSC3125Tuesday morning we tried something new during the General Session. Howard Eisenberg gave an informative presentation on item writing best practices, and at certain points he would pause so that each table could discuss the current topic amongst ourselves. Then he would take comments from some of the tables before moving forward with the next topic. The conversations at the table where I was sitting were GREAT! The sharing of different perspectives and experiences resulted in a lot , “Oh, I never thought of that!” expressions all around the room. THAT’S why I love going to our users conferences:  there’s just nothing like the information exchange and growth that takes place when a bunch of Questionmark users gather together in one place.

If you weren’t able to make it to Barcelona, I hope you can come to San Antonio, Texas, March 4 – 7. We’ll be right on the city’s River Walk. I’ve been there several times visiting family in the area and, I can tell you it’s beautiful and FUN! Please join us. I’m confident that by the end of the conference you will agree that it was time well spent.

What will you take to San Antonio?

Joan Phaup HeadshotPosted by Joan Phaup

river walk

Riverwalk, San Antonio

We’ve had an enthusiastic response to our announcement of San Antonio as the location for the Questionmark 2014 Users Conference March 4 – 7. We’ll be at the Grand Hyatt, right next to the famous Riverwalk and within easy walking distance of many local attractions.

But the most important thing on our minds is the conference program. Results of a recent customer survey will help us provide content that addresses current concerns and questions about assessment-related best practices, as well as the use of Questionmark technologies and services.

Equally significant will be the content created by Questionmark users themselves — people who present case studies or lead discussions.

We are  seeking case study and discussion proposals from now until October 18, so consider what you’d like to contribute. Please note that presenters will receive some red carpet treatment — including a special dinner in their honor on Tuesday, March 4th. And we award one 50% registration for each case study presentation.

Now is a good time to ask yourself what you could bring to the conference:

doug teaching 2013

2013 Users Conference

  • An account of how you are using assessments to support organizational goals?
  • Some lessons learned and advice for others?
  • A unique application of online or mobile assessments?
  • An account of how you have integrated Questionmark with another system?
  • A topic you think would be worth discussing with colleagues?

Click here for more details and proposal forms. Even if you are not sure you’ll attend the conference, we would like to hear from you! And whether you plan to present or not, plan now to have the conference in your budget for 2014. You will find information about conference return on investment and an ROI tookit here.

Crab cakes, customers and cool code in Charm City

John Kleeman HeadshotPosted by John KleemanHarbor View

I’ve just back from the Questionmark user conference in Baltimore “Charm City”, famous for its delicious crab cakes.

The best crab cakes let the flavour of crab shine through the other ingredients that hold the cake together. Similarly, with a conference, the main ingredient is the learning we take home with us. The other things that hold the event together – the networking, social events and discussions that take place – make the learning very enjoyable.

Here are six trends I observed last week:

1. Scaling up works well and is commonplace. Many Questionmark users have high volumes without drama. One case study at the conference told of a company deploying Questionmark with SAP in two months to deliver safety qualification assessments to 35,000 employees. And another told of delivering 2.3 million Questionmark assessments in 2012.

2. Authoring improvements. Our recent web-based authoring improvements are exciting users. In the 25 years we’ve been operating, Questionmark software has always been about enabling users to easily create, deliver and report on tests, quizzes, surveys and exams. There is a lot of excitement about what we’ve introduced recently in web-based authoring of questions and assessments, especially the way people can easily collaborate and see and review changes in items.

3. Acceptance of the Cloud. Some organizations prefer to deploy on-premise, others deploy in the Cloud with Questionmark OnDemand, and we support both. With other HR systems moving to the Cloud, more Questionmark customers are looking at moving to OnDemand. One influence is that SAP is telling all its HR customers that the future for them is in the Cloud with SuccessFactors; people are thinking “if you’re using SAP in the Cloud, then it makes sense to use Questionmark there, too”.charles networking

4. 70:20:10 is a given, and people are learning how to use it. Charles Jennings keynote was not without its controversies – many people still believe in the Kirkpatrick and Phillips models of learning evaluation, but he suggests levels 1, 2 and 5 have dubious validity in today’s learning environment. But there is widespread acceptance that the 70:20:10 model makes sense in corporate learning – that we learn 70% of what we learn on the job, 20% from others and only 10% from formal learning. There are questions around how you assess workplace learning – observational assessments are many people’s answer, and how you combine 70:20:10 with mandatory compliance training and ensuring people learn to work safely.

5. New methods of secure delivery matter. The availability of Questionmark Secure for use with Macs is making it easier for organizations, especially universities/colleges, to use a secure browser for delivering exams. People are also starting to use remote proctoring via organizations like our partner, ProctorU. Mobile assessment delivery is also now real not just being experimented with.

6. Cool code makes a difference. There’s been a lot of talk in recent years about how HTML 5 will make a difference to user interfaces and it’s great to see this happening for real. Questionmark is using HTML 5 to make our authoring and delivery interfaces easier and more natural. See for example this video: How to create hotspot questions for quizzes and tests. Code like this is exciting our customers and is going to provide very compelling web interfaces moving forwards.

I hope you enjoyed this and our other blog entries about the conference. We do our best to share about Questionmark and assessment remotely, but if you have a chance to attend one of our US or European user conferences in person, I promise you will learn and gain value.

Better testing drives better instruction — An update from the Questionmark Users Conference

Joan Phaup HeadshotPosted by Joan Phaup

My headline came from yesterday’s pre-conference workshop on Critierion-Referenced Test Development led by Sharon Shrock and Bill Coscarelli.

Sharon’s brief remark is a fitting theme for what’s happening this week in Baltimore at the Questionmark Users Conference, as assessment and measurement professionals come together to make testing and assessment better than ever — and as a result doing so much to improve learning.

photo (5)

Last night’s dessert reception brought Questionmark customers and staff back together again for our 11th annual conference.

The opening general session gave us a preview of what’s to come during some of the concurrent sessions, which will include:

  • bring-your-own laptop instruction in the use of Questionmark Live browser-base authoring
  • demonstrations of  Questionmarks’ new OData API for Analytics
  • sessions on incorporating mobile delivery into assessment programs
  • customer case studies
  • a session on best practices for leveraging SharePoint in a learning infrastrucure
  • a presentaton about secure testing in remote environments
  • a brainstorming session about possibilities for ADL’s new Experience API

We are looking forward to tomorrow’s keynote address by Charles Jennings of the 70:20:10 Forum on Meeting the Challenge of Measuring Informal and Workplace Learning.

This is a special conference for us, as we are also celebrating Questionmark’s 25th anniversary! We’re excited to be learning and celebrating with customers, and we look forward to these next few days. You can follow the conference and learn some new things on Twitter at #QMCON, so check in whenever you like.

The Portalization of eLearning and Assessment

Joan Phaup HeadshotPosted by Joan Phaup

Social networking, wikis, blogs, portals and collaboration tools offer powerful ways to increase participation and sustain momentum in learning communities – but how can these tools be blended to create strong learning ecosystems?  Enterprise applications such as Microsoft SharePoint can play a powerful role here, through content management and information sharing.

I talked about this the other day about this with Bill Finegan, who is Vice President – Enterprise Technology Solutions at GP Strategies. He is an expert on  information systems and operations management, with a heavy focus on learning technology.

Bill will be explaining Best Practices for Leveraging SharePoint in your Learning Infrastructure at the Questionmark Users Conference March 3 – 6 in Baltimore. His presentation will help people who are trying to map out their strategy for the evolution of their learning technology as well as technical professionals who are interested in discussing how to link systems together.

How are enterprise portal applications changing the learning landscape?

Bill Finegan

Bill Finegan

What we’re seeing in larger organizations is the need to move to what I tend to call a composite application approach (a mash-up), where the different enterprise applications are being linked to from one landing site, one location, one major hub.

This is allowing learners to get to their learning from the company portal site, one consolidated spot. It’s allowing organizations to tie learning into other enterprise applications such as  SharePoint and enterprise Learning Applications like SuccessFactors, SumTotal, and Questionmark — something they haven’t been able to in the past.

I’ve heard you refer to the “portalization” of eLearning. Could you talk about that development?

We’ve seen  it from two angles. One is the Learning Management Systems attempting to make themselves into that overall portal. You’ll see such as interfaces as the latest version s of SuccessFactors, SumTotal, Moodle, etc. They may have a portal look and feel with links out to different applications.

That being said, a lot of our customers are layering products like SharePoint on top of the LMS to have easier operability. Suppose they were using SuccessFactors as the LMS and Questionmark for assessments. They would leverage SharePoint as their intranet. If they had an existing intranet site, they would allow a subpage for learning, to give a more design-centered approach. With most LMSs going to a software-as-a service (Saas) environment, the portal allows a more personalized look and feel while not interfering what you are doing from SaaS perspective  (no customizations, etc.) and allowing for cleaner upgrades and so on.

How can organizations makes sense of all these different possibilities?

By deciding on their approach to collaboration and their overall approach to social learning. Are they looking for a Facebook-type approach?  An Amazon-type approach?  What systems do they want to use, and how do they want to connect the dots? If they have five different systems but don’t want to go to five different pages, how do they want users to get where they need to go?  Do they want to integrate through their LMS? Of do they want to put a portal on as the interface to their systems and use it to provide discussion threads and other collaboration tools?

How do you envision SharePoint, Questionmark and other systems working effectively together?

I view it as allowing for Questionmark functionality to get linked from and applied at a “presentation-level” perspective from SharePoint for notifications of available assessments — and to allow Questionmark to be the assessment engine underneath the portal. The portal would work the same way with other applications. The main learning technology applications become the proverbial “engine underneath the hood,” powering the systems in place but allowing for a more flexible and intuitive interface.

What would you like your audience to take away?

That Questionmark is ready for portalization, that portalization fits in with integrating Questionmark cleanly with other learning technical applications and that Questionmark and SharePoint can fit together in an overall mash-up/composite application approach.

You can learn more at the conference about this and choose from more than 30 other presentationsRegister online today!

Five Steps to Better Test Design and Delivery

Doug Peterson

Posted by Joan Phaup

I’ve been enjoying a series of posts in this blog my colleague Doug Peterson about Test Design and Delivery – so much so that I suggested he elaborate on this theme during a presentation at the Questionmark 2013 Users Conference in Baltimore March 3 – 6 – and he said, “Yes!”

Doug will be presenting on some other topics, too, but during a recent conversation with him I asked if he could tell me a little more about this particular session, which will focus on five processes:

1. Plan: Establish your test’s reliability and validity, and identify content areas to be covered
2. Create: Write items that increase the cognitive load, avoid bias and measure what’s important
3. Build: Pull items together into a test form, develop clear instructions and set passing scores
4. Deliver: Protect test content, control item exposure, protect test content and discourage cheating
5. Evaluate: Use item-, topic-, and test-level data to assess reliability and improve quality

Could you talk about your own background as a test author?

It mainly stems from what I was doing in the 3 or 4 years before I joined Questionmark. My group was responsible for training call center employees, and that included writing and administering lots of tests. Before my group took that over, all the tests were paper-and-pencil and had to be graded by the instructors. And of course you know that instructors over a period of time tend to bond with their students and tend to lose their objectivity.

It was clear that subjective testing was not good! We were introduced to Questionmark and we said, “Let’s automate these tests and make sure they are objective and fair.” That’s when I really got heavy-duty into testing. Over the course of those few years I attended several Questionmark conferences and went to a number sessions on item analysis, test analysis, setting cut scores, and so forth. I tried to understand all those kinds of things so that we could run statistical reports on our own content and sure our tests were valid and reliable.

Those years were very full of testing, and I learned a great deal about item and test writing, secure delivery and analyzing test results. I applied everything I learned to our call center training tests, and the customer satisfaction numbers began to rise. Why? Because our tests were working! The tests were valid and reliable, and because of that, they were weeding out the people who truly were not qualified for the job. Our stakeholders were very pleased because they had confidence that our tests were only passing people who were qualified to work in the call centers.

What do you think are the most challenging aspects of test design and delivery?

That’s hard to answer, because there are so many important things to think about! At the end of the day, the main thing is that the assessment is fair to both the test taker and the stakeholder, That idea encompasses many, many things. For the stakeholder, it requires having a valid, reliable assessment that uses solid methodology. For the participant, it boils down to well-written items. That sounds pretty simple, but it actually requires careful attention to detail.

How will you be addressing these challenges during your presentation at the Users Conference?

We’re going take a look at everything I’ve been working on in the blog series. What does reliable mean? What does valid mean? How can we appropriately plan an assessment and tie it back to the job or subject matter we’re testing for? We will also incorporate a lot of ideas about item writing. Throughout the session, we will be looking at fairness to the stakeholder and fairness to the participant and breaking those principles down into several components.

Who would benefit from attending this session?

Anyone who has anything to do with creating and delivering assessments: item writers, assessment assemblers, administrators. It’s good for people in these different roles to understand the entire test development and delivery process, so they appreciate their co-workers’ concerns. I see this session as suitable for people who are just beginning their work with Questionmark as well as those at the intermediate level. I’m looking forward to sharing so much of what I learned when I was so closely involved in a testing program myself.


There’s a lot to learn at this conference! Check out the agenda – and save $100 if you register by January 18th, 2013