Dr. David Metcalf, keynote speaker

Posted by Julie Chazyn

I’m writing today from the 2010 Questionmark Users Conference  in downtown Miami.  We kicked off last night with a poolside reception at the Hiton Miami Downtown and have a full day of case studies, technical training, best practice presentations and discussions ahead of us.

Enjoying the Sunday night Session at the 2010 Users Conference in Miami

We are looking forward to Tuesday’s general session, where our keynote speaker, Dr. David Metcalf of the University of Central Florida’s Institute for Simulation and Training will discuss Assessments on the Move: Mobility, Mashups and More.

Check out some of the photos from today and keep up with what’s happening via Twitter in real time. Look for the #qm10 hash tag to find out what’s going on at the conference!

The following podcast, recorded when we were planning the Users Conference, provides some hints about what David will be covering in tomorrow’s keynote:

Posted by Joan Phaup

Neelov Kar

Neelov Kar, Project Management Program Owner for Dell Services (previously Perot Systems) is getting ready to attend the Questionmark Users Conference in Miami this month. He will be delivering a case study about how he and his team have used statistical analysis to improve their test questions. I spent some time talking with Neelov the other day and wanted to share what I learned from him.

Q: Tell me a little about your company.

A:  We are a one-stop shop for IT Services and have people working all over the world, in 183 countries.

Q: What does your work entail?

A: I’m the project management program owner, so I am in charge of all the project management courses we offer. I help identify which courses are appropriate for people to take, based on training need analysis, and I work with our project management steering committee  to work out what courses we need to develop. Then we prioritize the requirements, design and develop the courses, pilot them and finally implement them as a regular course. As a Learning and Development department we also look after leadership courses and go through a similar process for those. I moved into this role about a year ago. Prior to that I was leading the evaluation team, and it was during my time on that team that we began using Questionmark.

Q: How you do you use online assessments?

A: We use Questionmark Perception for Level 2 assessment of our project management and leadership courses. We started with a hosted version of Questionmark Perception and it was I who actually internalized the tool. We offer leadership courses and project management courses internally within the organization across all geographies. Some of the project management courses already had tests, so we converted those to Questionmark.  We started designing the end-of-course assessments for our newly introduced leadership and project management courses once we started using Perception.

Q: What you will be talking about during your conference presentation?

A:  Last year we introduced a new course named P3MM Fundamentals, and because it was a new course we had to pilot the course with some of our senior members. In the pilot we asked the students to take the end-of-course test, and we found that many people had trouble passing the test. So we analyzed the results and refined the questions based on the responses. Analysis of results within Perception — particularly the Assessment Overview Report, Question Statistics Report, Test Analysis Report and Item Analysis Report — helped us in identifying the bad questions. We also saw that there were things we could do to improve the instruction within the course in order to better prepare people for the test. Using the Questionmark reports, we really perfected the test. This course has been going for over a year now, and it’s pretty stable. Now, every time we launch a course we do a pilot, administer the test and then use the Questionmark tools to analyze the questions to find out if we are doing justice to the people who are taking the test.

Q: What are you looking forward to at the conference?

A: I want to find out what Perception version 5 offers and how we can use it for our benefit.  Also, I saw that there are quite a few good papers to be presented, so I’m looking forward to attending those. And I want to get involved in the discussion about the future of SCORM.

Neelov’s is just one of 11 case studies to be presented at the conference, which will also include technical training, best practice presentations, peer discussions and more. Online registration for the conference ends on Tuesday, March 9th, so if you would like to attend, be sure to sign up soon!

Posted by Joan Phaup

Nine is today’s magic number — the 9th of March, that is: the last day to register online for the Questionmark Users Conference March 14 – 17th in Miami.

.

Here are a few more significant numbers for you as we count down to March 14th:

Learn. Share. Empower.

• 11 Customer Case Studies including two about integrations of Questionmark Perception with SAP
• 8 Tech Training presentations including Planning Your Migration from Perception v4 to v5
• 6 Tech Central sessions where Perception users meet one-on-one with a Questionmark Technician and get their questions answered
• 6  Product Central sessions, including focus groups run by Questionmark product managers seeking customers’ views on potential new features
• 6 Best Practices presentations including one on Item and Test Analysis Analytics
• 3 Peer Discussions including one on Security Strategies for High-stakes Tests
• 3 Drop-in Demos featuring Questionmark Live, new features in Perception version 5, mobile delivery and more
• 3 exciting Evening Events, including a dinner cruise with drinks, dinner and networking opportunities
• 2 General Sessions where Questionmark Customers learn about new and future technologies
• 1 Keynote by Dr. David Metcalf of the University of Central Florida’s Institute for Simulation and Training about Assessments on the Move: Mobility, Mashups and More

If you’re still wondering whether to attend or not, check out what participants in last year’s conference have to say in this video!

We are pleased to announce that we will be holding a one-day symposium in Berlin, Germany this April, which will provide the opportunity for our German Users to gather and learn about the latest developments from Questionmark and best practices in online assessment management. This year’s event will highlight the new features available in Questionmark Perception version 5, as well as providing technical training and case studies from experienced Questionmark Users. Questionmark Symposia are also a great opportunity to network and learn from other users and speak directly with Questionmark senior management. If you’d like to attend, make sure you sign up today and take advantage of the early-bird discounts!

http://www.questionmark.com/deu/seminars/symposium.aspx

The symposium is one of many Questionmark user events that take place around the world each year. As I write, many Questionmark Users are preparing to gather in Miami, Florida, for the three-day Questionmark Users  Conference. Online registration for the conference, which includes case studies, technical training, sessions on best practices, peer discussions and many other learning activities, ends on March 9th.

Registration for our U.K. Breakfast Briefings is now open! They will take place on three days in three cities: May 11 in Manchester, May 12 in Edinburgh and May 13 in London.

Breakfast Briefings are a great way to learn about online assessment technology and find out about the newest technologies and services  from Questionmark. They’re also a great opportunity to meet other assessment professionals in your area and talk with Questionmark staff.

This year we’ll be sharing the latest from Questionmark, including:

• How to create learning mashups by embedding quizzes in wikis and portals
• Using auto-sensing and auto-sizing to deliver assessments to mobile devices
• Creating items with Questionmark Live
• Reporting and analytics
• Plus much more!

If you’re new to online assessment this is a great opportunity to see the technology in action, and talk with Questionmark staff about the best solution for your organisation. If you’re already a Questionmark customer come along to join us for breakfast and see recently released features and those that are coming soon! Click here to sign up for Manchester, London or Edinburgh.

The program for the Questionmark Users Conference in Miami March 14 -17 offers something for every Questionmark Perception user, whether they are just starting out or have been using Perception for many years. Here’s the complete line-up of conference sessions.

The conference will also include opportunities to meet one-on-one with Questionmark product managers and technicians, and to attend networking events.

General Sessions

• Opening General Session: Conference kick-off
• Keynote: Dr. David Metcalf on “Assessments on the Move: Mobility, Mashups and More”
• Closing General Session: The Road Ahead

Customer Case Studies

• Integrating Questionmark Perception and SAP for Employment Testing
• Streamlined Training and Assessment at NPPD: Integrating Perception with SAP Learning Solution
• Using Flash and Captivate in Questionmark Perception
• Questionmark Live: Distributing the Workload and Increasing Accessibility
• Many from One: Creating Hundreds of Questions to QML Import Using Excel
• Question Bank as a Knowledge Tree: Sowing the Seeds of Knowledge
• Questionmark in Dental Education: Why, How and Lessons Learned
• Establishing a Broad National Certification Network Based on Questionmark
• Questionmark Perception as a Central Component of a Training Strategy
• Perfecting the Test Through Question Analysis
• The Cloud School: Learning and Assessment for Underserved People in Brazil

Tech Training

• Introduction to Questionmark Perception for Beginners
• Advanced Authoring Techniques Using Authoring Manager
• Planning Your Migration from Perception v4 to v5
• Configuring the User Experience and Understanding Templates in Perception v5
• Analyzing and Sharing Results with Enterprise Reporter
• Using Perception for Evaluations and Other Types of Surveys
• Methods for Integrating Perception with Other Systems

Best Practices

• Item and Test Analysis Analytics
• Making Your Assessments Available and Accessible to All
• Questionmark Live Item Writing Workshop: Bring Your Own Laptop!
• Technical Standards and Questionmark’s Open Assessment Platform
• Conducting Validity Studies within Your Organization
• Effectively Measuring Knowledge, Skill and Ability with Well-crafted Assessments

Peer Discussions:

• Strategies for High-Stakes Tests
• Shaping the Future of ADL SCORM: What’s On Your Wish List?
• Using Questionmark To Facilitate ROI strategies

Drop-in Demos

• Options and More Options: Different Ways to Deliver Your Assessments
• What’s New in Perception Version 5?
• Tools/Features to Enhance Your Use of Perception

Register for the conference and join us in Miami!

Q: What’s your role at Questionmark?

Steve Lay

A: I’m the Integration Team Lead, so I look after all aspects of the product  concerned with integrating Questionmark with other systems — both within an organization and over the Internet.  I am Product Owner for our Connector products, which enable integrations of Questionmark Perception with other key systems such as Sharepoint Portal Server, Blackboard, Moodle and others. And I am heading our Open Assessment Platform Initiative.

Q: You just mentioned the Open Assessment Platform. How would you describe that?

A: We’re creating a software platform as a way of making Questionmark the perfect complement to other systems people have within their organizations. We are documenting, in an open way, how you can interact with Questionmark software at both the Web service and Web development level to make integration projects go smoothly and easily.

Q: What standards will you be discussing during your Best Practices session?

A:  I am going to be talking about some of the standards in development now that we are are looking at as having the potential for support in the future.  The IMS Global Learning Consortium, for example, are in the process of publishing an interesting specification about how learning tools can be put together – learning tools interoperability.  I will also be talking about standards for exchanging data about people, IMS Learning Information Services and specifications from HR-XML — and the possibilities they open up for synchronizing people and organizational information with Perception. I’m also be going to talk about some of the more generic standards that might be having an impact on us too, such as XHTML 5 and the emerging standards for authentication and authorization such as OpenID and OAuth.  And of course, I’ll be touching on SCORM, too, although I will also be co-facilitating a separate discussion devoted entirely to the future of SCORM with Daniel Rehak from Advanced Distributed Learning.

Q: How do you see people using the information they learn in your session?

A: I hope people get a deeper understanding of what’s happening in some of these developments and take that back into their own organizations in order to figure out the impact the standards will have on them. As well as providing feedback to Questionmark, I’d like them to feel able to talk about their requirements with all their suppliers and understand how they can influence the ongoing standards process. I hope people take a way more than just the acronyms and what they stand for – I’d like them to get some ideas about how they might start conversations within their own organizations to help them prepare for future adoption.

Q:  What are you looking forward to most at the conference?

A: Of course, like every Product Owner, I’m looking forward to meeting all the new and returning customers!  It will be good opportunity to talk in a little more depth with people about integration issues, how they are thinking of using the new version of Perception and what opportunities our Open Assessment Platform presents. I have to add that I’m looking forward to a bit of sunshine as well!

Learn more about the many  conference activities by visiting www.questionmark.com/go/conference. And register soon to attend the conference, which will take place March 14 – 17 in Miami.

John Kleeman

Making Your Assessments Accessible and Available to All will be presented by Performance Solutions Specialist Cheryl Johnson and Questionmark Chairman John Kleeman during the Questionmark Users Conference March 14 – 17 in Miami. Cheryl brings a strong commitment to accessibility and usability to her work as an instructional designer as well as a trainer. She has been inspired by seeing how technology can dramatically expand opportunities for individuals with disabilities. I am looking forward to seeing Cheryl at the conference and would like to share my recent conversation with her:

Q: Could you tell me about your work as an instructional designer?

Cheryl Johnsoon

A: I’ve been involved with many  corporate and government training programs over the years. Although I have developed some instructor-led training, I’ve worked primarily on elearning. Lately I’ve been moving into social learning, mobile learning, high-level simulations, and gaming.

Q: Let’s talk specifically about the subject of accessibility: could you discuss your experience with that?

A: In the mid-nineties I started training people to use assistive technologies. That was before assistive technology was really considered a productivity tool. Back then it was only about making information and technology accessible to people. There really were no tutorials or any good training out there, so I started writing my own training with the motivation of reducing the amount of tech support I had to provide to the people I’d trained! A few years later I was living in Utah and often had to train people remotely. My learners were primarily vocational rehabilitation clients, and sometimes I would  be training just one person at a time. It was not cost-effective for me to drive long distances to train one person. I worked with a colleague to develop what would have been called in those days a “distance learning” program to help people use voice recognition technology effectively.  He went on to patent the technology, called VoiceWindows (http://www.voiceteach.com/), as an online tutorial. It is a tutorial on voice recognition technology and its use with various software applications.  In addition there are many macros built into it to increase productivity when using voice recognition technology. I have also trained many quadriplegics and have seen the huge impact technology could have on the quality of their lives, in some cases opening up work opportunities. Technology can open up a lot of possibilities and help make people be more productive both inside and outside the workplace.

Q: What are the key things people involved with assessment need to know about accessibility?

A: It calls for being creative and thinking differently about things. When I teach classes on accessibility I explain some rules: things like 508 compliance and the various standards people need to meet. But people get frustrated because they don’t know how to meet those requirements using what they have. Section 508 compliance really is just the basic standard. It’s more about making sure  the person can access that information rather than whether is it really usable. I try to focus on helping people make information and technology usable as well as accessible. There are no written rules out there for doing that, so creativity is key. I recommend that when organizations are designing assessments or learning materials, they have people who use the technology on the design team from the very beginning. They know their technology. They know how things work. And they can suggest alternative ways of achieving something. That might mean taking a hot spot question, which would require the use of a mouse, and making text links for the hot spot areas so that someone who’s unable to use a mouse can answer the same question using a keyboard.

Q: What do you hope people will take away from your presentation?

A: An understanding of how important it is to make sure that people who use assistive technology are part of their design team. A realization that quite often people who use assistive technology are sometimes novice users and need clear direction and instructions in using it for a particular application—for example a  particular assessment that you have adapted for their use. People with disabilities like to use interactive tools and want as rich a learning and assessment experience as everyone else.  And of course,  enthusiasm for finding creative ways to make assessments more enjoyable, usable and effective for everyone. John will be showing how Questionmark Perception addresses accessibility and usability issues. He’ll be covering what you have to do to make your assessments in Perception accessible, so that will be a major takeaway, too!

Q: What are you looking forward to at the Questionmark Users Conference?

A: I’m excited to meet people who want to make high quality learning and assessment available to everyone. That’s exciting to me, because I’ve spent many years fighting that battle! I also have to admit I’m looking forward to the cruise on Tuesday night!

Join Cheryl, John and our many other presenters for three days of learning and networking. Check out the conference program and register today!

Posted by Joan Phaup

Steve Lay

Daniel R. Rehak, a technical advisor to the Advanced Distributed Learning Initiative (ADL), is responsible for technical and strategic advice primarily about the direction and refinement of ADL’s activities in the development of SCORM and content repositories and registries. As part of its plan to release an enhanced, harmonized version of its SCORM interoperability standards in 2011, ADL is actively seeking the views of SCORM and AICC community members. Dan is coming to the Questionmark Users Conference, takinig place in Miami March 14 – 17,  to hear what customers hope to see in an updated version of SCORM. He and Questionmark Integration Team Lead Steve Lay will co-facilitate a discussion on Shaping the Future of ADL SCORM:  What’s On Your Wish List?

I recently asked Dan about his ADL work and his desire to sit down with Questionmark users to discuss the future of SCORM:

Q: How would you define SCORM?

Daniel Rehak

A: SCORM is a model — a collection of different interoperability standards that for the most part were originally created to work on their own. SCORM explains how to make them work together with the objective of taking learning activities, managed by a learning management system, and making them portable, interoperable and reusable across different vendors and technology platforms. In addition to being used by many commercial organizations and the military, it’s used for K12 education in the UK and Korea. It’s used widely all over the world, notably in higher education and industry.

Q: Could you tell me about your role with ADL SCORM?

A: I’m involved in planning the ADL SCORM update: looking at the form the update will take, who we should talk to, the timeline, the tasks we need to do,  the technical direction, and then communicating those out to various people: vendors, users, and their organizations both here in the US and internationally. I’m also helping ADL upgrading its registry federation model for learning content.

Q: Can you share some details about the planned update to SCORM?

A:  SCORM has not been through a major update since 2004, although we made some minor, incremental changes along the way.  By talking with vendors and user communities, we are finding divergent views about what’s important and what’s lacking — those kinds of things. We know that technology has evolved on the Web.  We also know we have legacies out there: we have people procuring SCORM-based delivery systems today with the expectation that they’re going to be around in a number of years and that they are still going to work. So the objective is to produce an update that is grounded in the reality of what people want to use and what they actually have in place. We have to balance legacy support requirements with new capabilities and requirements. Our goal is to produce an updated, revised and harmonized version of SCORM by late 2011. We are talking to users, talking to adopters and to vendors to find out what things they are planning to put into their products in the short term – say the next 18 months or so. We may not be able to do everything we want to do in that  timeframe, but  our assumption is that things that don’t hit this cycle will get into the next cycle.

Q: What issues are you finding as you talk with people about the future of SCORM?

A: A number of things keep coming up: Web services, different content exchange models, content mark-up languages, assessments – those are the kinds of things that are on the list. What we are trying to do is develop a flexible model that indicates what we feel is important and that vendors can get behind. We also want organizations to be able to add features of their own in harmony with what we are doing. We know we won’t be able to do everything ourselves; we are relying on the community to bring their resources to bear on this and help us collectively move things forward.  We call it harmonization because we want to bring people together and move forward together.

Q: What’s prompted you to attend the Questionmark Users Conference?

A: We’re reaching out to a number of different vendors to find out what their users are interested in. We’ve been in discussions with Questionmark for awhile and this came up as an opportunity to talk with this user segment. Assessments standards are not currently in SCORM, but we know assessments and tracking are things people have said are very important. You have one of the best user communities there to talk to, so we want to talk with them and find out what they see as being important in SCORM as we go forward.

Q: Who should attend this session?

A: Anyone from the SCORM community.  We’ve got different communities that have different levels of adoption. Also, anyone who uses AICC, which has some elements in common with SCORM. They have a significant overlap in what they do and how they do it. We have been in direct communication with AICC about the harmonization and would welcome ideas from their users. Also, from the standpoint of specifications and standards, we’re interested in hearing from anyone with opinions about IMS QTI.

Q: What kind of input are you looking for during your session about the future of SCORM?

A: We want to know what people are currently doing; what problems they’re seeing; what they think in general about assessment and tracking. We’d like to learn more about where they do and don’t use SCORM and why. We’ll also ask what we can change in SCORM to make their lives better.

Come to the conference and let your voice be heard! You can register online or email conference@questionmark.com for further information.