What makes technology-enabled learning and assessment different in the rest of world? Language instantly springs to mind when we consider what sets one country apart from another. But other differences need to be considered, too, when deciding how best to use technology- enabled learning and assessments.

A recent post in Questionmark CEO Eric Shepherd’s blog explores the differences that arise when you cross social, economic and geographical boundaries.  Eric poses the question: Apart from Language, What Challenges Make Technology Enabled Learning and Assessment Different in the Rest of the World? He then identifies four key points that might drive us to use different kinds of assessments depending on where we are in the world:

1. Invalid Assumptions About Internet Connectivity (Internet connectivity will create dramatically different experiences for a student in the Amazon and a student in the USA or Europe)
2. Cost of Internet Device – The cost of purchasing a computer or PDA  in Europe or North America represents a fraction of an average annual salary, whereas in some areas of the world the cost might be 6 – 12 months of an average person’s salary. The resulting use of smaller, lower cost, generally mobile devices in poorer areas of the world calls for the re-sizing of content to accommodate them.
3. Conformance with Local Laws: Laws regarding data privacy, accessibility and equal access vary from country to country.
4. Culture: Contrasting value systems can cause different cultures to think differently about such aspects of assessment as cheating. Cheating may be thought of as solidarity within a culture that promotes collectivism and loyalty. 

For more insights on this subject and many others, visit Eric’s blog at http://blog.eric.info.

I’d like to draw your attention to some recent writing by our CEO, Eric Shepherd, on the subject of “Assessment Security and How To Reduce Fraud.”

In this post on his blog, Eric describes  what motivates fraudulent behavior and explains some common methods and processes to help you minimize and eliminate it from your assessments. He references the “Fraud Triangle” created by famed criminologist Donald Cressey to explain why people commit fraud. Eric examines the three elements in the triangle: Motivation, Rationalization and Opportunity. He also offers some practical tips for deterring fraud.

If you are interested in this and many other issues surrounding assessment be sure to check out Eric’s blog.

As you probably know, I live in Miami, but as I spend more than 250 nights in hotels around the world I love it when I can stay at home. This year, I am happy that the Questionmark Users Conference is in Miami March 14 – 17! So much so that I’ve posted some pictures and videos to my blog that I feel capture the spirit of Miami, South Beach and what you can expect when you attend the conference.

This year’s conference will have a strong focus on Questionmark Perception version 5 and new trends in assessment delivery.  In addition to tech training, case studies, best practice sessions and peer discussions, you’ll be able to meet one-on-one with our technicians and product managers and network with other Perception users who share your interests.

We are proud to be making this conference greener than ever. The Hilton Miami Downtown participates enthusiastically in the Florida Green Lodging Program, and we at Questionmark are doing our part for the environment by reducing the amount of paper we use, decreasing the size of our conference notebooks and making handouts available online.

I love living in Miami and think it’s a vibrant, colorful and friendly backdrop for the conference. I hope you will take the opportunity to stroll down Lincoln Road in South Beach during our Monday evening dine-around and that you’ll join us on Tuesday evening for a dinner cruise around the harbor. These events are a lot of fun, but they also have a serious purpose: helping you get to know colleagues who share your interest in assessment and create connections that will help you succeed in your work.

I can’t wait to welcome you to the conference—and I am looking forward to learning together with you. So come on down to Miami and warm up!

Early-bird registration for the conference ends today, so I hope you can sign up soon.

This is the second of a series of blog posts on assessment standards. Today I’d like to focus on the IMS QTI (Question and Test Interoperability) Specification.

It’s worth mentioning the difference between Specifications and Standards: Specifications are documents that industry bodies have agreed on (like IMS QTI XML), while Standards have been published and committed to by a formal legal body (like AICC or HTML). A Specification is less formal than a Standard but still can be very useful for interoperability.

Questionmark was one of the originators of QTI. When we migrated our assessment platform from Windows to the Web in the 1990s, our customers had to migrate their questions from one platform to the other. As you will know, it takes a lot of time to write high quality questions, and so it’s important to be able to carry them forward independently of technology. We knew that we’d be improving our software over the years and we wanted to ensure the easy transfer of questions from one version to the next. So we came up with QML (Question Markup Language), an open and platform-independent method of maintaining questions that makes it easy for customers to move forward in the future.

Although QML did solve the problem of moving questions between Questionmark versions, we met many customers who had difficulty bringing content created in another vendor’s proprietary format  into Questionmark. We  wanted to help them, and we also wanted to embrace openness and allow Questionmark customers to export out their questions in a standard format if they ever wanted to leave us. So we worked with other vendors within the umbrella of the IMS Global Learning Consortium to come up with QTI XML, a language that describes questions in a technology-neutral way.  I was involved in the work defining IMS QTI as were several of my colleagues: Paul Roberts did a lot of technical design, Eric Shepherd led the IMS working group that made QTI version 1, and Steve Lay (before joining Questionmark) led the version 2 project.

Here is a fragment of QTI XML and you can see that it is a just-about-human-readable way of describing a question.

I’d like to draw your attention a great blog post by our CEO, Eric Shepherd on “Learning Mashups.”

In the post , Eric describes three types of Learning mashups, which combine different components (such as videos and embedded assessments) to create easy-to-maintain, stimulating and powerful learning environments. The post includes a couple of YouTube videos that demonstrate the practicality of mashups. Check this out on Eric Shepherd’s blog.

Posted by Eric Shepherd

I have been watching the Questionmark Blog with interest and thought that, as Questionmark’s CEO, it was about time that I made a contribution!

The Questionmark Blog was started to keep you in touch with our products, our news releases, learning materials and our Product Owners’ points of view.  We’ve been focusing on articles that assist assessment practitioners and instructional designers; recently we previewed how embedding syndicated assessments within wikis, web pages and blogs can support the learning process.

Separate to this initiative I have been running a personal blog (http://blog.eric.info) to bring you more abstract thoughts, observations from travels, and distillations of conversations that I’ve enjoyed along the way.  Not surprisingly the Tag Cloud quickly shows what I blog about, Assessments, Books, Travel and Questionmark.  Here are some links that you might find interesting:

•    Recent article on Learning Environments that explains how systems are now being built around Single Sign-on Portals, Wikis, Blogs and Data Warehouses
•    Questionmark Live – Story Behind the Story
•    Assessments Fundamentals with articles on Fidelity of an Assessment, Blooms Taxonomy, Item Analysis,   Types Of Assessments (Formative, Diagnostic, Summative, and Surveys), and many more.
•    A couple of YouTube videos, one titled Assessment as they relate to Learning Professionals
•    My Favorite Books , which relate to mostly to best practices in management and assessments. I’ll be posting more as I get time.

I look forward to meeting you out in the web 2.0 world!

Posted by Julie Chazyn

In creating good, solid surveys, quizzes, test and exams it’s essential to understand what type of assessment will give you appropriate and actionable results.  We believe the ultimate objective of the assessment directly influences how it will be structured. This requires understanding the subtle distinctions that can mean big differences in the quality and outcomes of your assessments.  The language we use in talking about assessments needs to reflect those distinctions.

With that in mind, Questionmark CEO Eric Shepherd recently took some time to update Questionmark’s UK and US glossaries to help people understand different types of assessments.

Some of the terms that have been altered include:

Diagnostic assessment
Personality assessment
Pretest
Psychological assessment
Summative assessment

We hope you will bookmark the glossary and refer back to it often!

Posted by John Kleeman

Questionmark has been a long time supporter of QTI XML. QTI XML is a language that describes  questions in XML in a way that is platform and technology neutral.

Questions take a long time to write, and it’s important that they can survive in computerized form as technology changes. At Questionmark, our customers often need to take questions constructed in an older system, sometimes a discontinued system, and move them into Questionmark software. When the legacy questions are in QTI XML, this is reasonably straightforward. When the questions are held deep inside a proprietary database or other structure, it can be more challenging to move them into another system.

Eric Shepherd, Paul Roberts and myself were part of the original IMS team that created QTI XML, and Eric led the team that created and evangelized it for many years. Steve Lay, before he joined Questionmark was one of the leaders of the QTI version 2 initiative. Questionmark as a company is a keen supporter of QTI XML – it’s a good way for our customers to be confident that questions are likely to survive into the long term future.

Version 1.2.1 of the IMS QTI specification was finalized in 2003 and is supported by Questionmark Perception and in our Content Converter application .

IMS has been working for many years on a version 2 of IMS QTI, and version 2.0 was introduced and finalized, and then a draft version 2.1. You may have seen some controversy recently as the IMS has withdrawn the draft version 2.1 of QTI, removing it from their website saying that they want to review it and change it.

Version 2.1 of QTI XML has been in draft form for a couple of years, and although Questionmark hasn’t yet supported it, very many projects have done so, and there is concern from those who have implemented it that their work might not be valid when 2.1 or 2.2 is finalized. See Rowin Young of CETIS’s blog for one perspective on this.

I’d like to reassure our customers that Questionmark continues to support QTI XML version 1.2.1 and will expect to support later versions of QTI XML when formally released as final and open specifications.

Posted by Tom King

There’s been much discussion among developers and insiders about recent posts and vendor notices regarding automated cheating tools for SCORM 1.2/SCORM 2004 content. I’d like to share some thoughts on the underlying issues and measures that can be taken to reduce vulnerability. This is no joke… unlike the recent fun on Eric’s blog.  This is the first in a series of posts on this topic. I’d like to review a few key concepts before delving into specific areas and solutions that will come in the posts to follow.

Security is important. The cornerstone of successful education, training and certification is the effective use of assessments. Inaccurate, misleading or falsified performance data can lead to poor decisions, increased liability and other significant risks or consequences. Questionmark is keenly aware of the importance of security and has addressed the issue from the beginning.

Vulnerabilities exist. Note that this is vulnerabilities in the plural. About a week ago, an independent elearning developer published one: a simple bookmarklet to send false score and status data to a SCORM LMS (see Cheating in SCORM, Phillip Hutchison, 2009). [Note: The post remains, but the sample bookmark has been removed after an outside party requested this.]

Using the published exploit is as simple as saving a book mark to your browser and then picking that bookmark while content is running. It is what I’ve called a “cheatlet” and current implementations may foreshadow other potential issues (see Security Before Features, Tom King, 2008 on LETSI SCORM 2.0 web site). Others have discussed how common in-browser debugging tools like Firebug can be used to similar effect. The key message is that this type of exploit is possible, and it gets easier and more viable over time.

Defense in depth. That phrase is a bit of a mantra in the cybersecurity world. Questionmark has taken this approach with its implementation of interoperability solutions, including SCORM. Some in the SCORM community recognize the “cheatlet” exploit as a known weakness that has just become easier for the common man to use. They go on to indicate that SCORM shouldn’t be used for high stakes assessments, and end their argument or response there. It is left for future specifications to deal with this issue. However there are several alternatives to decrease vulnerability.

I think I now have your attention for the subsequent posts. Also, if you’re attending the Questionmark User Conference 2009 this coming week in Memphis, please feel free to  stop by for my session on standards and ask about this or other standards-related issues.