We are pleased to announce that we will be holding a one-day symposium in Berlin, Germany this April, which will provide the opportunity for our German Users to gather and learn about the latest developments from Questionmark and best practices in online assessment management. This year’s event will highlight the new features available in Questionmark Perception version 5, as well as providing technical training and case studies from experienced Questionmark Users. Questionmark Symposia are also a great opportunity to network and learn from other users and speak directly with Questionmark senior management. If you’d like to attend, make sure you sign up today and take advantage of the early-bird discounts!

http://www.questionmark.com/deu/seminars/symposium.aspx

The symposium is one of many Questionmark user events that take place around the world each year. As I write, many Questionmark Users are preparing to gather in Miami, Florida, for the three-day Questionmark Users  Conference. Online registration for the conference, which includes case studies, technical training, sessions on best practices, peer discussions and many other learning activities, ends on March 9th.

What makes technology-enabled learning and assessment different in the rest of world? Language instantly springs to mind when we consider what sets one country apart from another. But other differences need to be considered, too, when deciding how best to use technology- enabled learning and assessments.

A recent post in Questionmark CEO Eric Shepherd’s blog explores the differences that arise when you cross social, economic and geographical boundaries.  Eric poses the question: Apart from Language, What Challenges Make Technology Enabled Learning and Assessment Different in the Rest of the World? He then identifies four key points that might drive us to use different kinds of assessments depending on where we are in the world:

1. Invalid Assumptions About Internet Connectivity (Internet connectivity will create dramatically different experiences for a student in the Amazon and a student in the USA or Europe)
2. Cost of Internet Device – The cost of purchasing a computer or PDA  in Europe or North America represents a fraction of an average annual salary, whereas in some areas of the world the cost might be 6 – 12 months of an average person’s salary. The resulting use of smaller, lower cost, generally mobile devices in poorer areas of the world calls for the re-sizing of content to accommodate them.
3. Conformance with Local Laws: Laws regarding data privacy, accessibility and equal access vary from country to country.
4. Culture: Contrasting value systems can cause different cultures to think differently about such aspects of assessment as cheating. Cheating may be thought of as solidarity within a culture that promotes collectivism and loyalty. 

For more insights on this subject and many others, visit Eric’s blog at http://blog.eric.info.

Registration for our U.K. Breakfast Briefings is now open! They will take place on three days in three cities: May 11 in Manchester, May 12 in Edinburgh and May 13 in London.

Breakfast Briefings are a great way to learn about online assessment technology and find out about the newest technologies and services  from Questionmark. They’re also a great opportunity to meet other assessment professionals in your area and talk with Questionmark staff.

This year we’ll be sharing the latest from Questionmark, including:

• How to create learning mashups by embedding quizzes in wikis and portals
• Using auto-sensing and auto-sizing to deliver assessments to mobile devices
• Creating items with Questionmark Live
• Reporting and analytics
• Plus much more!

If you’re new to online assessment this is a great opportunity to see the technology in action, and talk with Questionmark staff about the best solution for your organisation. If you’re already a Questionmark customer come along to join us for breakfast and see recently released features and those that are coming soon! Click here to sign up for Manchester, London or Edinburgh.

The program for the Questionmark Users Conference in Miami March 14 -17 offers something for every Questionmark Perception user, whether they are just starting out or have been using Perception for many years. Here’s the complete line-up of conference sessions.

The conference will also include opportunities to meet one-on-one with Questionmark product managers and technicians, and to attend networking events.

General Sessions

• Opening General Session: Conference kick-off
• Keynote: Dr. David Metcalf on “Assessments on the Move: Mobility, Mashups and More”
• Closing General Session: The Road Ahead

Customer Case Studies

• Integrating Questionmark Perception and SAP for Employment Testing
• Streamlined Training and Assessment at NPPD: Integrating Perception with SAP Learning Solution
• Using Flash and Captivate in Questionmark Perception
• Questionmark Live: Distributing the Workload and Increasing Accessibility
• Many from One: Creating Hundreds of Questions to QML Import Using Excel
• Question Bank as a Knowledge Tree: Sowing the Seeds of Knowledge
• Questionmark in Dental Education: Why, How and Lessons Learned
• Establishing a Broad National Certification Network Based on Questionmark
• Questionmark Perception as a Central Component of a Training Strategy
• Perfecting the Test Through Question Analysis
• The Cloud School: Learning and Assessment for Underserved People in Brazil

Tech Training

• Introduction to Questionmark Perception for Beginners
• Advanced Authoring Techniques Using Authoring Manager
• Planning Your Migration from Perception v4 to v5
• Configuring the User Experience and Understanding Templates in Perception v5
• Analyzing and Sharing Results with Enterprise Reporter
• Using Perception for Evaluations and Other Types of Surveys
• Methods for Integrating Perception with Other Systems

Best Practices

• Item and Test Analysis Analytics
• Making Your Assessments Available and Accessible to All
• Questionmark Live Item Writing Workshop: Bring Your Own Laptop!
• Technical Standards and Questionmark’s Open Assessment Platform
• Conducting Validity Studies within Your Organization
• Effectively Measuring Knowledge, Skill and Ability with Well-crafted Assessments

Peer Discussions:

• Strategies for High-Stakes Tests
• Shaping the Future of ADL SCORM: What’s On Your Wish List?
• Using Questionmark To Facilitate ROI strategies

Drop-in Demos

• Options and More Options: Different Ways to Deliver Your Assessments
• What’s New in Perception Version 5?
• Tools/Features to Enhance Your Use of Perception

Register for the conference and join us in Miami!

Q: What’s your role at Questionmark?

Steve Lay

A: I’m the Integration Team Lead, so I look after all aspects of the product  concerned with integrating Questionmark with other systems — both within an organization and over the Internet.  I am Product Owner for our Connector products, which enable integrations of Questionmark Perception with other key systems such as Sharepoint Portal Server, Blackboard, Moodle and others. And I am heading our Open Assessment Platform Initiative.

Q: You just mentioned the Open Assessment Platform. How would you describe that?

A: We’re creating a software platform as a way of making Questionmark the perfect complement to other systems people have within their organizations. We are documenting, in an open way, how you can interact with Questionmark software at both the Web service and Web development level to make integration projects go smoothly and easily.

Q: What standards will you be discussing during your Best Practices session?

A:  I am going to be talking about some of the standards in development now that we are are looking at as having the potential for support in the future.  The IMS Global Learning Consortium, for example, are in the process of publishing an interesting specification about how learning tools can be put together – learning tools interoperability.  I will also be talking about standards for exchanging data about people, IMS Learning Information Services and specifications from HR-XML — and the possibilities they open up for synchronizing people and organizational information with Perception. I’m also be going to talk about some of the more generic standards that might be having an impact on us too, such as XHTML 5 and the emerging standards for authentication and authorization such as OpenID and OAuth.  And of course, I’ll be touching on SCORM, too, although I will also be co-facilitating a separate discussion devoted entirely to the future of SCORM with Daniel Rehak from Advanced Distributed Learning.

Q: How do you see people using the information they learn in your session?

A: I hope people get a deeper understanding of what’s happening in some of these developments and take that back into their own organizations in order to figure out the impact the standards will have on them. As well as providing feedback to Questionmark, I’d like them to feel able to talk about their requirements with all their suppliers and understand how they can influence the ongoing standards process. I hope people take a way more than just the acronyms and what they stand for – I’d like them to get some ideas about how they might start conversations within their own organizations to help them prepare for future adoption.

Q:  What are you looking forward to most at the conference?

A: Of course, like every Product Owner, I’m looking forward to meeting all the new and returning customers!  It will be good opportunity to talk in a little more depth with people about integration issues, how they are thinking of using the new version of Perception and what opportunities our Open Assessment Platform presents. I have to add that I’m looking forward to a bit of sunshine as well!

Learn more about the many  conference activities by visiting www.questionmark.com/go/conference. And register soon to attend the conference, which will take place March 14 – 17 in Miami.

John Kleeman

Making Your Assessments Accessible and Available to All will be presented by Performance Solutions Specialist Cheryl Johnson and Questionmark Chairman John Kleeman during the Questionmark Users Conference March 14 – 17 in Miami. Cheryl brings a strong commitment to accessibility and usability to her work as an instructional designer as well as a trainer. She has been inspired by seeing how technology can dramatically expand opportunities for individuals with disabilities. I am looking forward to seeing Cheryl at the conference and would like to share my recent conversation with her:

Q: Could you tell me about your work as an instructional designer?

Cheryl Johnsoon

A: I’ve been involved with many  corporate and government training programs over the years. Although I have developed some instructor-led training, I’ve worked primarily on elearning. Lately I’ve been moving into social learning, mobile learning, high-level simulations, and gaming.

Q: Let’s talk specifically about the subject of accessibility: could you discuss your experience with that?

A: In the mid-nineties I started training people to use assistive technologies. That was before assistive technology was really considered a productivity tool. Back then it was only about making information and technology accessible to people. There really were no tutorials or any good training out there, so I started writing my own training with the motivation of reducing the amount of tech support I had to provide to the people I’d trained! A few years later I was living in Utah and often had to train people remotely. My learners were primarily vocational rehabilitation clients, and sometimes I would  be training just one person at a time. It was not cost-effective for me to drive long distances to train one person. I worked with a colleague to develop what would have been called in those days a “distance learning” program to help people use voice recognition technology effectively.  He went on to patent the technology, called VoiceWindows (http://www.voiceteach.com/), as an online tutorial. It is a tutorial on voice recognition technology and its use with various software applications.  In addition there are many macros built into it to increase productivity when using voice recognition technology. I have also trained many quadriplegics and have seen the huge impact technology could have on the quality of their lives, in some cases opening up work opportunities. Technology can open up a lot of possibilities and help make people be more productive both inside and outside the workplace.

Q: What are the key things people involved with assessment need to know about accessibility?

A: It calls for being creative and thinking differently about things. When I teach classes on accessibility I explain some rules: things like 508 compliance and the various standards people need to meet. But people get frustrated because they don’t know how to meet those requirements using what they have. Section 508 compliance really is just the basic standard. It’s more about making sure  the person can access that information rather than whether is it really usable. I try to focus on helping people make information and technology usable as well as accessible. There are no written rules out there for doing that, so creativity is key. I recommend that when organizations are designing assessments or learning materials, they have people who use the technology on the design team from the very beginning. They know their technology. They know how things work. And they can suggest alternative ways of achieving something. That might mean taking a hot spot question, which would require the use of a mouse, and making text links for the hot spot areas so that someone who’s unable to use a mouse can answer the same question using a keyboard.

Q: What do you hope people will take away from your presentation?

A: An understanding of how important it is to make sure that people who use assistive technology are part of their design team. A realization that quite often people who use assistive technology are sometimes novice users and need clear direction and instructions in using it for a particular application—for example a  particular assessment that you have adapted for their use. People with disabilities like to use interactive tools and want as rich a learning and assessment experience as everyone else.  And of course,  enthusiasm for finding creative ways to make assessments more enjoyable, usable and effective for everyone. John will be showing how Questionmark Perception addresses accessibility and usability issues. He’ll be covering what you have to do to make your assessments in Perception accessible, so that will be a major takeaway, too!

Q: What are you looking forward to at the Questionmark Users Conference?

A: I’m excited to meet people who want to make high quality learning and assessment available to everyone. That’s exciting to me, because I’ve spent many years fighting that battle! I also have to admit I’m looking forward to the cruise on Tuesday night!

Join Cheryl, John and our many other presenters for three days of learning and networking. Check out the conference program and register today!

Making assessments and questions translatable was a key goal for Questionmark Perception version 5.

There always has been a need to translate assessments, particularly in countries like Switzerland and Canada, which have have different languages within their borders. But the Internet has made the world much more connected, and so many organizations have employees and stakeholders in different countries or speaking different languages. Some typical translation needs that fed into version 5 were:

• A bank that operates in many parts of the world and wants to deliver course evaluation surveys (level 1s) to participants in training in many languages. It’s crucial for them that they can schedule the assessment and allow their participants to choose the language in which the assessment is taken
• A telephone company that trains employees in Europe and wants to give similar questions to all employees. They want to author the questions once and then translate them for use by training teams in each country.
• A university that creates and delivers questions in English but is expanding internationally, is in partnership with universities overseas and wants to translate some of its material into other languages
• A software company that deliver certifications to consultants and partners worldwide in around 20 languages and needs assessments and questions translated so that the certifications can be given fairly to anyone who speaks the languages they support.
• A manufacturing support company that authors questions in English and then translates them into 8 European and Asian languages for delivery to their partners and employees worldwide.

Some customers use external translators and want Questionmark to export XML that translators can use in specialist tools, whilst other customers have internal expertise and want to translate in house and need screens within Questionmark Perception to make the translations themselves. For all customers, it’s not just the initial translation that they need help with but the management of the process. When questions change, if you have them translated into a dozen language, it’s a nightmare to keep track manually of what needs updated, and Questionmark software needs to flag when a question has changed and remind you to update the translation.

The basic concept of Questionmark’s translation management system is that you create a question in a base language and then translate it into as many target languages as you want. And then you do the same with an assessment.

Questions and assessments are linked so that you can report across language.

And to do the translation, you can translate in Perception by typing in the text in a simple user interface shown below.

Translating within Questionmark Perception works well if you have an in-house translator, but if you are working with external translators, it’s usually best to export to XML and send them the text to translate – and they will return it to you to import into Perception. Text is exported in an industry standard XML called XLIFF that standard software packages used by translators can process.

Whether you translate interactively or by export to XML, Questionmark keeps track of when questions change in the base language and prompts you to update translations to keep translated questions up to date. So we help you not just manage the initial translation process, but also the ongoing process of translations as questions and assessments change and evolve.

We’re very excited about how easy it is to translate questions in Perception version 5, and we look forward to your feedback as you use it to create quizzes, tests, exams and surveys that can be used in many languages.

Posted by Joan Phaup

Steve Lay

Daniel R. Rehak, a technical advisor to the Advanced Distributed Learning Initiative (ADL), is responsible for technical and strategic advice primarily about the direction and refinement of ADL’s activities in the development of SCORM and content repositories and registries. As part of its plan to release an enhanced, harmonized version of its SCORM interoperability standards in 2011, ADL is actively seeking the views of SCORM and AICC community members. Dan is coming to the Questionmark Users Conference, takinig place in Miami March 14 – 17,  to hear what customers hope to see in an updated version of SCORM. He and Questionmark Integration Team Lead Steve Lay will co-facilitate a discussion on Shaping the Future of ADL SCORM:  What’s On Your Wish List?

I recently asked Dan about his ADL work and his desire to sit down with Questionmark users to discuss the future of SCORM:

Q: How would you define SCORM?

Daniel Rehak

A: SCORM is a model — a collection of different interoperability standards that for the most part were originally created to work on their own. SCORM explains how to make them work together with the objective of taking learning activities, managed by a learning management system, and making them portable, interoperable and reusable across different vendors and technology platforms. In addition to being used by many commercial organizations and the military, it’s used for K12 education in the UK and Korea. It’s used widely all over the world, notably in higher education and industry.

Q: Could you tell me about your role with ADL SCORM?

A: I’m involved in planning the ADL SCORM update: looking at the form the update will take, who we should talk to, the timeline, the tasks we need to do,  the technical direction, and then communicating those out to various people: vendors, users, and their organizations both here in the US and internationally. I’m also helping ADL upgrading its registry federation model for learning content.

Q: Can you share some details about the planned update to SCORM?

A:  SCORM has not been through a major update since 2004, although we made some minor, incremental changes along the way.  By talking with vendors and user communities, we are finding divergent views about what’s important and what’s lacking — those kinds of things. We know that technology has evolved on the Web.  We also know we have legacies out there: we have people procuring SCORM-based delivery systems today with the expectation that they’re going to be around in a number of years and that they are still going to work. So the objective is to produce an update that is grounded in the reality of what people want to use and what they actually have in place. We have to balance legacy support requirements with new capabilities and requirements. Our goal is to produce an updated, revised and harmonized version of SCORM by late 2011. We are talking to users, talking to adopters and to vendors to find out what things they are planning to put into their products in the short term – say the next 18 months or so. We may not be able to do everything we want to do in that  timeframe, but  our assumption is that things that don’t hit this cycle will get into the next cycle.

Q: What issues are you finding as you talk with people about the future of SCORM?

A: A number of things keep coming up: Web services, different content exchange models, content mark-up languages, assessments – those are the kinds of things that are on the list. What we are trying to do is develop a flexible model that indicates what we feel is important and that vendors can get behind. We also want organizations to be able to add features of their own in harmony with what we are doing. We know we won’t be able to do everything ourselves; we are relying on the community to bring their resources to bear on this and help us collectively move things forward.  We call it harmonization because we want to bring people together and move forward together.

Q: What’s prompted you to attend the Questionmark Users Conference?

A: We’re reaching out to a number of different vendors to find out what their users are interested in. We’ve been in discussions with Questionmark for awhile and this came up as an opportunity to talk with this user segment. Assessments standards are not currently in SCORM, but we know assessments and tracking are things people have said are very important. You have one of the best user communities there to talk to, so we want to talk with them and find out what they see as being important in SCORM as we go forward.

Q: Who should attend this session?

A: Anyone from the SCORM community.  We’ve got different communities that have different levels of adoption. Also, anyone who uses AICC, which has some elements in common with SCORM. They have a significant overlap in what they do and how they do it. We have been in direct communication with AICC about the harmonization and would welcome ideas from their users. Also, from the standpoint of specifications and standards, we’re interested in hearing from anyone with opinions about IMS QTI.

Q: What kind of input are you looking for during your session about the future of SCORM?

A: We want to know what people are currently doing; what problems they’re seeing; what they think in general about assessment and tracking. We’d like to learn more about where they do and don’t use SCORM and why. We’ll also ask what we can change in SCORM to make their lives better.

Come to the conference and let your voice be heard! You can register online or email conference@questionmark.com for further information.

This is the fourth of a series of  posts on standards that impact assessment.

The ADL SCORM standard rose out of an initiative by the US Department of Defence (DoD), who were large users of e-learning. They wanted to ensure that e-learning content could be interoperable and reusable, for instance to ensure that if e-learning content was developed by one vendor, it could run in another vendor’s environment.

The DoD has a track record of setting technical standards:  for instance  in the 1980s they helped popularize and make TCP/IP an effective standard. The DoD is also a very large customer for most companies in the learning and technology software industry. So when the DoD announced that it would only purchase e-learning software that worked with SCORM, the industry jumped quickly to support it!

One of the ways in which SCORM was made successful was by a series of Plugfests, where vendors could get together in practical labs and check that interoperability was possible in practice, not just in theory. These were well run events, a kind of technological speed dating, where each vendor could try out their compatibility with other vendors. It was great to have technical experts from each vendor in the room and be able to have many different LMSs all able to call our assessments.

In Questionmark Perception, to make an assessment run via SCORM, you use the Publish to LMS capability to create a content package, which is a small XML document that references the assessment. And as you can see in the screenshot below, you can choose from AICC and two flavours of SCORM. Once you’ve made the package, you simply upload it to a management system and participants can then be directed to it.

SCORM is used widely, both within the military and outside it. If you have a choice between AICC and SCORM, it’s often better to choose AICC (see my earlier post in this series), partly because SCORM has a potential security issue (see our past blog article). However, providing you are aware of this issue, SCORM can be a very effective means of calling assessments.

The ADL are currently reviewing SCORM and working out how to improve it, including potentially making it more useful for assessments. As part of their listening for this review, ADL’s technical advisor, Dan Rehak, who was one of the architects of SCORM, is running a session at Questionmark’s user conference in Miami in March to gain feedback on how SCORM could be improved. If you’re interested in influencing this standard to be better in future, this would be a great session to go to. Stay tuned here on the blog for a Questionmark Conference Close-up interview with Dan.