Dr. David Metcalf, keynote speaker

Posted by Julie Chazyn

I’m writing today from the 2010 Questionmark Users Conference  in downtown Miami.  We kicked off last night with a poolside reception at the Hiton Miami Downtown and have a full day of case studies, technical training, best practice presentations and discussions ahead of us.

Enjoying the Sunday night Session at the 2010 Users Conference in Miami

We are looking forward to Tuesday’s general session, where our keynote speaker, Dr. David Metcalf of the University of Central Florida’s Institute for Simulation and Training will discuss Assessments on the Move: Mobility, Mashups and More.

Check out some of the photos from today and keep up with what’s happening via Twitter in real time. Look for the #qm10 hash tag to find out what’s going on at the conference!

The following podcast, recorded when we were planning the Users Conference, provides some hints about what David will be covering in tomorrow’s keynote:

In a previous blog post, I explained the Brinkerhoff Success Case Method (SCM), which is a way of conducting training evaluation by finding examples of success or failure. You then do to the people who have experienced success or failure, and use these to identify the potential of a new training program. It’s particularly effective to identify early on whether a new initiative is successful and what its potential is.

A key step in the SCM is that you need to identify the participants who’ve been successful, usually via a survey. Questionmark Perception provides a great way of sending such a survey that participants can respond to quickly and easily which identifies how successful they’ve been in applying the training course in their jobs.

For example, following a training program to train someone in a new tool, you might name some possible business applications of the tool, and then ask a 4-point Likert scale question on each application, perhaps:

Have you used the training to get some specific benefit

• Yes and had clear and concrete positive results
• Yes but no clear results yet
• To some extent but don’t expect any results
• Have not applied the training in this area at all

Participants who give the first choice on a few different applications are candidates for success interviews, and candidates who answer the last two choices are candidates for failure interviews. The main purpose of the survey is to identify success and failure candidates for interview, but a secondary purpose can be to look at the overall results and see how successful overall the program has been as rated by participants.

This is an example of a Perception assessment to ask questions like this.

Pretend you are a respondent who answers it, and you’ll see whether you are a candidate for an interview. You can take it more than once by clicking on Next Assessment at the end of one attempt. Such an assessment could be delivered inside a wiki or blog posting like this, or pushed to respondents via an email link or other system.

In comparison to other methods of evaluation, the Success Case Method is very quick and inexpensive to do, and seems an interesting way of evaluating training.  And Questionmark seems a great way of delivering SCM surveys.

I’d welcome feedback from anyone who tries the Success Case Method with Questionmark software.

We are happy to announce  the availability of a free Questionmark iPhone/ iPod touch app on iTunes that opens up new possibilities for delivering knowledge checks, quizzes  and surveys to a wide, mobile audience: a great way to engage learners on the go!

The auto-sensing and auto-sizing capabilities of Questionmark™ Perception™ version 5 makes this type of app possible by streamlining and simplifying the delivery of surveys, quizzes and tests to multiple devices, in multiple contexts.

Try out the Questionmark iPhone/iPod touch app to take some Questionmark Perception assessments on your iPhone or iPod touch. Test your knowledge of world facts and geography with our “Round the World” or “State Capitals” quizzes or try your hand at several other topics. Imagine how this new delivery option could help your organization, and let us know what you think! (You will need to have iTunes installed on your computer and download the app from the Apple app store.)

Users of Questionmark Perception version 5 can have their own personalized apps so that their employees and/or students can access assessments, quizzes or surveys on the go.
For more information on how to get your personalized app, contact customercare@questionmark.com.

Embed a Questionmark Perception assessment, survey or quiz inside your blog entries on WordPress.

• To see how this would look, see embedded quizzes within our own blog.

• Check out this How-to on our developer Web site.

• WordPress is an open source blog publishing application and can be used for basic content management. A low-stakes quiz or survey within WordPress is a great way to interact with your blog and collect information from your readers.

Please note it is only possible to include the IFrame code in a self-hosted WordPress blog.

I’ve been reading recently about the Success Case Method for training evaluation, and would recommend  Robert Brinkerhoff’s book, The Success Case Method, as an intriguing read.

The Success Case Method (SCM) is a way of identifying how successful a training program or other initiative for change is. Using the SCM you identify particularly successful participants in the training program, and interview them to determine why they were successful and write these up as stories to illustrate where the program can be successful. By highlighting the experience of the most successful participants, and writing up their story in a way that others can connect with, the SCM allows people to see what success means for the training program and identify what needs to be improved for others to get the success. Sometimes you also interview people who represent failures who did not get any success out of the training program, to understand qualitatively why.

By doing a qualitative interview, you can dig into the real reasons for something. And also by focusing on what is the very best, the most successful that the training program can be, you can identify what can work. And you can do a SCM very quickly – as it says on the cover of the book “Find out quickly what’s working and what’s not”.  The SCM also seems a practical way of doing “level 3” like assessments, in terms of seeing how behavior is impacted. And because it’s easy to do and need not be comprehensive to be useful, it is quick and inexpensive.

The five steps in the method are

1. Plan a Success Case study
2. Create an “impact model” which defines what success should look like, i.e. what improvements you are looking for
3. Design and implement a survey to participants to search for best and worst cases
4. Interview and document the success stories
5. Communicate findings and recommendations

The Brinkerhoff Success Case Method looks a very interesting and cost-effective way of identifying success factors for a training initiative.

I’ll follow up this post with some ways to use Questionmark Perception to do a Success Case Study.

Hear from John and other Questionmark managers at the Questionmark Users Conference March 14- 17 in Miami. Online registration ends tomorrow, March 9th. Check out the full conference schedule and plan to be in Miami for this important learning event.

Conduct a survey or ask questions during or after an online presentation in Live Meeting.

• Check out this How-to on Questionmark’s developer Web site.

• Microsoft Office Live Meeting is a hosted Web conferencing service that connects people in online meetings, training, and events. Incorporating a Questionmark Perception assessment or survey into a Live Meeting can help keep your colleagues engaged and provide an easy way to gather feedback about your presentation in real time.

Posted by Joan Phaup

Neelov Kar

Neelov Kar, Project Management Program Owner for Dell Services (previously Perot Systems) is getting ready to attend the Questionmark Users Conference in Miami this month. He will be delivering a case study about how he and his team have used statistical analysis to improve their test questions. I spent some time talking with Neelov the other day and wanted to share what I learned from him.

Q: Tell me a little about your company.

A:  We are a one-stop shop for IT Services and have people working all over the world, in 183 countries.

Q: What does your work entail?

A: I’m the project management program owner, so I am in charge of all the project management courses we offer. I help identify which courses are appropriate for people to take, based on training need analysis, and I work with our project management steering committee  to work out what courses we need to develop. Then we prioritize the requirements, design and develop the courses, pilot them and finally implement them as a regular course. As a Learning and Development department we also look after leadership courses and go through a similar process for those. I moved into this role about a year ago. Prior to that I was leading the evaluation team, and it was during my time on that team that we began using Questionmark.

Q: How you do you use online assessments?

A: We use Questionmark Perception for Level 2 assessment of our project management and leadership courses. We started with a hosted version of Questionmark Perception and it was I who actually internalized the tool. We offer leadership courses and project management courses internally within the organization across all geographies. Some of the project management courses already had tests, so we converted those to Questionmark.  We started designing the end-of-course assessments for our newly introduced leadership and project management courses once we started using Perception.

Q: What you will be talking about during your conference presentation?

A:  Last year we introduced a new course named P3MM Fundamentals, and because it was a new course we had to pilot the course with some of our senior members. In the pilot we asked the students to take the end-of-course test, and we found that many people had trouble passing the test. So we analyzed the results and refined the questions based on the responses. Analysis of results within Perception — particularly the Assessment Overview Report, Question Statistics Report, Test Analysis Report and Item Analysis Report — helped us in identifying the bad questions. We also saw that there were things we could do to improve the instruction within the course in order to better prepare people for the test. Using the Questionmark reports, we really perfected the test. This course has been going for over a year now, and it’s pretty stable. Now, every time we launch a course we do a pilot, administer the test and then use the Questionmark tools to analyze the questions to find out if we are doing justice to the people who are taking the test.

Q: What are you looking forward to at the conference?

A: I want to find out what Perception version 5 offers and how we can use it for our benefit.  Also, I saw that there are quite a few good papers to be presented, so I’m looking forward to attending those. And I want to get involved in the discussion about the future of SCORM.

Neelov’s is just one of 11 case studies to be presented at the conference, which will also include technical training, best practice presentations, peer discussions and more. Online registration for the conference ends on Tuesday, March 9th, so if you would like to attend, be sure to sign up soon!

Posted by Joan Phaup

Nine is today’s magic number — the 9th of March, that is: the last day to register online for the Questionmark Users Conference March 14 – 17th in Miami.

.

Here are a few more significant numbers for you as we count down to March 14th:

Learn. Share. Empower.

• 11 Customer Case Studies including two about integrations of Questionmark Perception with SAP
• 8 Tech Training presentations including Planning Your Migration from Perception v4 to v5
• 6 Tech Central sessions where Perception users meet one-on-one with a Questionmark Technician and get their questions answered
• 6  Product Central sessions, including focus groups run by Questionmark product managers seeking customers’ views on potential new features
• 6 Best Practices presentations including one on Item and Test Analysis Analytics
• 3 Peer Discussions including one on Security Strategies for High-stakes Tests
• 3 Drop-in Demos featuring Questionmark Live, new features in Perception version 5, mobile delivery and more
• 3 exciting Evening Events, including a dinner cruise with drinks, dinner and networking opportunities
• 2 General Sessions where Questionmark Customers learn about new and future technologies
• 1 Keynote by Dr. David Metcalf of the University of Central Florida’s Institute for Simulation and Training about Assessments on the Move: Mobility, Mashups and More

If you’re still wondering whether to attend or not, check out what participants in last year’s conference have to say in this video!

We are pleased to announce that we will be holding a one-day symposium in Berlin, Germany this April, which will provide the opportunity for our German Users to gather and learn about the latest developments from Questionmark and best practices in online assessment management. This year’s event will highlight the new features available in Questionmark Perception version 5, as well as providing technical training and case studies from experienced Questionmark Users. Questionmark Symposia are also a great opportunity to network and learn from other users and speak directly with Questionmark senior management. If you’d like to attend, make sure you sign up today and take advantage of the early-bird discounts!

http://www.questionmark.com/deu/seminars/symposium.aspx

The symposium is one of many Questionmark user events that take place around the world each year. As I write, many Questionmark Users are preparing to gather in Miami, Florida, for the three-day Questionmark Users  Conference. Online registration for the conference, which includes case studies, technical training, sessions on best practices, peer discussions and many other learning activities, ends on March 9th.