Seven deadly sins of item writing – with apologies to William Shakespeare

Posted by John Kleeman

“Double, double toil and trouble;
Fire burn, and caldron bubble.”The Three witches, by Johann Heinrich Fussli

So said the three witches in Shakespeare’s Macbeth as they prepared to lead the lord Macbeth to doom and destruction.

Today is Halloween and the evenings in London have been getting darker.  A few days ago, I was walking home through Hampstead Heath after work. The wind whispered through the trees, and there was thunder, lightning and rain in the air. I thought I heard witches’ voices in the distance. Was it a dream or was it real? I do not know, but here is what I can recall:

The witches were discussing how to muddle, befuddle or scuttle a testing programme. They suggested seven deadly sins of item writing:

1. Use lots of not. Don’t avoid negatives: indeed if it’s not too hard, include not a few double negatives not infrequently to make your test-takers struggle.

2. Trick your test-takers. Remember how we witches tricked Macbeth to murder the King and showed up his weaker qualities. Mislead your test-takers, add tricks into your questions, so only the truly strong will pass your tests.

3. Include speling and grammer errors. It’ll make your test takers feel more at ease if you include a few language misteaks in each question.

4. Never use one word where you can use 13. Make your question wording and choices as long as you can; that will sort out the toads from the  frogs.

5. We love “All of the above” and “None of the above”. They give so much room for confusion, delusion and illusion in multiple choice items.

6. Subtle, subtle is number six. Make sure your items test factual knowledge only; don’t test higher order skills like comprehension or application. Avoid scenarios or anything that checks real-world skills; measure as little useful capability as you can.

7. Most importantly, skip all the review. Having other people read and review your questions, fit them to a blueprint, check them for language, psychometrics and bias is so boring!  Be bloody, bold and resolute – deploy your questions without wasting time waiting for others’ opinions.

This is fiction of course. But fair is foul, and foul is fair – follow these seven rules if you want your tests to be unfair!

Assess to Comply: How else can you ensure that employees understand?

ECOAPosted by John Kleeman

Earlier this month, I gave a presentation on assessments and compliance at the Ethics & Compliance Officer Association’s (ECOA) 20th Annual Ethics & Compliance Conference in St Louis, Missouri.

ECOA is the professional association for and a leading provider of resources to ethics, integrity, and compliance practitioners worldwide. The conference gathered together ethics and compliance officer and managers from around the United States, and many other countries, for the exchange of best practices, strategies, and ideas.

One of the conference plenary sessions was in the form of a debate and asked whether the last 20 years of ethics and compliance has made a difference in the business world. Recent years have seen many huge regulatory fines, and even criminal penalties, with companies around the world falling afoul of ethical or legal guidelines. Would we all be better without an ethics and compliance industry? During his keynote address at the ECOA conference, Carlo di Florio (Director of the SEC Office of Compliance Inspections and Examinations), made his view clear, that from a regulatory perspective, government entities like the SEC really appreciate ethics and compliance departments, and the job of a regulator would be much more difficult without ethics and compliance programs in organizations.

I strongly believe that assessments can make a big difference in ethics and compliance. They measure knowledge/understanding of regulations and assess culture and ethics. Above all, online assessments are one of the few ways, perhaps the only way, in which a compliance team can touch on every employee in a large organization. You cannot speak personally to every employee – but you can ask every employee to take a test or answer a survey.

During my presentation, I suggested five key benefits of ethics and compliance assessments:

  • Demonstrate commitment
  • Find problems early
  • Document understanding after training
  • Increase knowledge / enhance retention
  • Reduce cost and time for compliance training

I also shared what I think are five key and actionable pieces of good practice for ethics and compliance assessments:

  • Create scenario questions
  • Use feedback at the topic level
  • Make assessments accessible
  • Have participants agree to a code of conduct
  • Consider observational assessments

You can see the presentation on SlideShare here or embedded below:

4 Tips for making your assessments BYOD-Friendly

Posted by Brian McNamara

Many organizations have begun to embrace the concept of “BYOD” (Bring Your Own Device), so we thought it would be useful to share a few tips on how to optimize your online assessments for the broadest range of devices and browsers possible.

Mobile devices are increasingly being used for delivering online surveys and quizzes. We’re also seeing more customers using mobile devices for observational assessment, and exploring the potential of tablets for “mobile test centers.”

Here are a few tips to keep in mind if you’re planning to deliver to smartphones or tablets:

1. Think small. Fortunately, Questionmark’s auto-sensing, auto-sizing interface makes it easy to accommodate a broad range of devices. However, you still should consider the word-count of your items and types of content you wish to deliver. For example, large images that are crucial to a question’s stimulus and/or choices could put users of small-screen devices at a disadvantage. Likewise, they may also take longer to load if the mobile device has a less-than-optimal data-connection signal.

2. Provide QR Codes to make it easy to access quizzes and surveys via mobile devices. A QR Code can contain a URL that makes it quick and easy to launch an assessment, improve survey response rates, and enable capturing of demographic data. See the blog article for more info: “Using QR Codes – Start to Finish

3. Be cautious using Flash, or avoid it altogether, as many devices (particular iOS devices such as the iPhone) do not provide native support for it.

4. Test it out! Try your assessments on as many different devices as practical. There are many “emulators” that you can use on PCs to help understand how content will appear on the ‘small screen’ — but be cautious as they don’t always give a true “user experience.”

If you’d like to learn more, sign up for one of our web seminars on mobile assessment delivery!

Assessments Through the Learning Process : Video

Posted By Doug Peterson

Assessments – surveys, quizzes, tests, and exams – do so much more than determine whether or not a learner passed a training course. In this video, we’ll take a look at the variety and importance of the roles played by assessments before, during and after a learning experience.

For more details, download this white paper: Assessments Through the Learning Process

How SAP use Questionmark to deliver certification exams in 20 languages

Posted by John Kleeman

At the recent European Association of Test Publishers conference, I presented jointly with Sue Martin and Ralf Kirchgaussner of SAP on blended delivery within SAP’s certification programme. You can see our presentation on SlideShare, but I thought I’d highlight in this blog how SAP uses Questionmark’s Translation Management System to organize its translation of certification exams in up to 20 languages.

SAP sees certification as an enabler for project success and quality. There is evidence in the IT industry (see for example here) that functional performance improves when a majority of a project team are certified. So for SAP, certification is part of their strategic need to ensure successful use of their technology. SAP are also a genuinely global company; and if it needs to be done, it needs to be done for people in a variety of geographies and languages.

Languages used in the SAP certification programmeAs you can see in the Languages graph on the right, the most popular language for SAP certifications is English. But many are delivered in other languages with German, Spanish, Portuguese and Japanese the most widely used non-English languages.

SAP translate items and assessments using Questionmark’s Translation Management System, exporting the text to XML for translation by professional translators. They then import the translated text back into the Questionmark Perception assessment management system for review and delivery.

See the screenshot below showing a topic in the SAP repository translated into 18 languages:

Questionmark Perception Translation Management System

I hope you enjoyed this glimpse into how SAP use translated assessments. You can see more about SAP’s certification programme and how they are looking to move into a more blended delivery environment in our presentation which is embedded below:

New online training options for Questionmark users

Questionmark’s European Training Manager, Kate Soper

Posted by Chloe Mendonca

People who want to get started using Questionmark but aren’t able to attend a regular three-day training course now have the option of taking online training.

In response to requests from our customers, we are pleased to offer a variety of short online courses that participants can take from the comfort of their own homes or offices. All of the courses are a mix of a live, interactive presentation where you’ll be encouraged to ask questions and a set of exercises and support materials (including short video tutorials) for you to work through at your own pace and schedule.

Three new online courses will be led by Questionmark’s European Training Manager, Kate Soper. Kate has been delivering Questionmark training courses since 2008 and had extensive experience in e-assessment before that. Course participants will benefit from Kate’s expert advice and instruction, as well as additional support after the courses have finished.

  • Introduction to Authoring Manager V5.5
    New Questionmark users will learn in this course how to use Authoring Manager to create topic folders, questions and assessments. (​Two to three hours, including a 15-minute break.)
  • Introduction to Enterprise Manager V5.5
    This course explains how to create and schedule participants and create other administrators. This course provides an overview of Questionmark’s functions, including the basics of reporting. (​Two to three hours, including a 15-minute break.)
  • Customising the Participant Interface in Version 5.5
    Learn how to tailor the look and feel of your assessments in this course. Do you want to add your logo to login screens and assessments? Change the ‘Submit’ button to say ‘Finish test’? Add a link to your terms and conditions on the login screen? Then this is the course for you! (Five to six hours in two separate sessions, each including a 15-minute break)

Click here for complete details, including course objectives, pricing, dates and registration