Seven deadly sins of item writing – with apologies to William Shakespeare

Posted by John Kleeman

“Double, double toil and trouble;
Fire burn, and caldron bubble.”The Three witches, by Johann Heinrich Fussli

So said the three witches in Shakespeare’s Macbeth as they prepared to lead the lord Macbeth to doom and destruction.

Today is Halloween and the evenings in London have been getting darker.  A few days ago, I was walking home through Hampstead Heath after work. The wind whispered through the trees, and there was thunder, lightning and rain in the air. I thought I heard witches’ voices in the distance. Was it a dream or was it real? I do not know, but here is what I can recall:

The witches were discussing how to muddle, befuddle or scuttle a testing programme. They suggested seven deadly sins of item writing:

1. Use lots of not. Don’t avoid negatives: indeed if it’s not too hard, include not a few double negatives not infrequently to make your test-takers struggle.

2. Trick your test-takers. Remember how we witches tricked Macbeth to murder the King and showed up his weaker qualities. Mislead your test-takers, add tricks into your questions, so only the truly strong will pass your tests.

3. Include speling and grammer errors. It’ll make your test takers feel more at ease if you include a few language misteaks in each question.

4. Never use one word where you can use 13. Make your question wording and choices as long as you can; that will sort out the toads from the  frogs.

5. We love “All of the above” and “None of the above”. They give so much room for confusion, delusion and illusion in multiple choice items.

6. Subtle, subtle is number six. Make sure your items test factual knowledge only; don’t test higher order skills like comprehension or application. Avoid scenarios or anything that checks real-world skills; measure as little useful capability as you can.

7. Most importantly, skip all the review. Having other people read and review your questions, fit them to a blueprint, check them for language, psychometrics and bias is so boring!  Be bloody, bold and resolute – deploy your questions without wasting time waiting for others’ opinions.

This is fiction of course. But fair is foul, and foul is fair – follow these seven rules if you want your tests to be unfair!

Looking back (and forward) to #qmcon

It’s three weeks since the Questionmark European Users Conference and I’m still following up on some of the conversations that started there. The presenters really help shape the theme of the conference: each session provides the starting point for the discussion that happens in the breaks, in the bars and on into our community spaces and discussion forums after the event.

An increasing part of this conversation is something often referred to as the back-channel. For me, the back-channel is a way that participants in an event can contribute to a wider conversation around the event’s themes using their mobile devices.  To help you grapple with the details of this concept, you might like to read this interesting blog post preparing participants for DevLearn 2011:  “What exactly is a back-channel?” I particularly like the comment from a speaker encouraging audience members to use their mobile devices to engage.

So how would you tune into the Questionmark European Users Conference back-channel? If you were using Twitter (and that is the easiest way) you would  just search for all tweets that include the special hash-tag #qmcon (‘hash’ is one of the many names for the # symbol on your keyboard).  Most Twitter monitoring tools make it easy to monitor your search continuously so that you can see new comments appearing in near real time.  To contribute to the conversation about this event, you would simply include the term #qmcon in your tweet.

Don’t be shy!  Of course, when you tweet publicly it might be recorded and preserved for all time (especially if you make particularly witty and incisive contributions) but back-channel comments are short and conversational and usually ephemeral like any other comment or question you ask in a discussion session. Why not give it go?  Next time you’re at a conference find out the back-channel hashtag and have a look.  If you’ve got a burning comment or question you don’t need to wait until the end of the session anymore: tweet it and see what comes back.

I always like to monitor the back-channel at every conference I attend, and when I’m presenting I always check back afterwards to see if there are comments and questions I need to follow up on or ways I can improve my session next time.  For example, here was a useful tip from someone in the audience of one of my sessions:

“#qmcon Acronym city in Integration session. developers have a language all of their own, no pun intended.”

Clearly I need to humanise that slide deck a bit for next time.

In my opinion, following hashtags is the way to use Twitter.  And it isn’t just something people do at conferences either.  Producers of live TV programmes monitor hashtags and provide feedback to the presenters in real time. You can also follow major trends and world events; try searching for something of interest and see what tags people are using to follow the conversation.

Still enjoying the Halloween spirit?  Just follow #pumpkin for a constant stream of pumpkin-related tweets and pictures.

13 Scary Questions to Ask your Assessment Cloud Provider

Posted by John Kleeman

As its Halloween I thought you might enjoy learning about 13 questions that might scare your Assessment Cloud provider.

Let me first share some background information …13 Scary Questions to Ask your Assessment Cloud Provider

With increasing use of Cloud systems like Google Docs, Microsoft’s Office 365, and Amazon, and with enterprise software giants like Oracle and SAP offering OnDemand services, many organizations that previously managed IT internally are delegating the running of servers. A Cloud service can save you money, and allow you to focus on core business and user issues, by letting someone else deal with the technology.

Secure and scalable Assessment Clouds are the next wave of tools available that help organizations to measure knowledge, skills, and attitudes securely for certification, regulatory compliance and successful learning outcomes
As you consider moving your assessments to a Cloud, you need to ensure your provider is offering the best possible service, security and data protection. You want a provider who is fully invested in giving strong security, scalability, elasticity and robustness, not just someone running a server under a desk! Exam security has different challenges and demands to other kinds of IT due to the confidentiality of personally identifiable information, questions and results, so you need to make sure that the system you use is safe and secure.

Here are 13 questions you might scare the less professional Assessment Cloud providers in the marketplace:

1. Do you host assessments in a well-established Data Center, certified to SAS 70 Type II, SSAE 16 Type II or ISO 27001?

2. Does your Data Center have multiple connections to the power grid with onsite generators with at least 24 hours fuel onsite in case of power outages?

3. Does your Data Center have multiple, fast Internet links so that if one goes down, connectivity remains?

4. Is every server in the system load balanced and does every component have redundancy, so that if any one system fails, another can take over?

5. Is browser access to assessments and administration protected by SSL (or TLS) to 128 bits or higher, so that assessment data and results cannot be intercepted on the Internet?

6. Do you follow industry good practice in software development to reduce surface areas of attack and protect against security vulnerabilities? Common methodologies to work with are called STRIDE and DREAD.

7. Do you have separate development/integration areas and staging areas to test on before deploying to production?

Questionmark’s OnDemand Testing and Deployment Process
Questionmark’s OnDemand Testing and Deployment Process

8. Do you have a data security policy for your employees who run the service to ensure that they maintain the secrecy of customer data? Does the policy include confidentiality agreements, background checks on employees, regular training, and regular testing of employees to check they that understand data security?

9. Can I see information on real time information on the current status and uptime, and access statistics from round the world? See status.questionmark.com for an example of what you might look for from a provider.

10. Is the service monitored and run 24/7 at both Data Center, network, hardware and application level, so that problems out of hours will be fixed?

11. Are results data backed up safely at least once an hour, so that in the event of a catastrophe, you should never lose more than an hour’s worth of data?

12. What access might government agencies have to data of foreign nationals and are your systems Safe Harbour Certified?

13. What is your track record do you have for being a trustworthy provider with references and case studies to back your claims up?

The answers to these questions for Questionmark’s OnDemand Service are all yes. If you want to find out more, read more details in our new white paper, Security of Questionmark’s OnDemand Service available here.