Why Questionmark for certification testing?

Posted by John Kleeman

I recently wrote about the business benefit of running certification programs, particularly in high-tech industries. Today, I’m happy to share some links to real-world examples of certification programs that use online assessments – and to explain a little about how our customers can author assessments which can be delivered in a variety of ways.

A few examples of successful online certification programs:

Annual certifications at an automotive supply company enable employees to demonstrate their knowledge of machinery, blueprints, safety practices, quality procedures and documentation. Switching to online certifications has enabled the company to replace subjective one-on-one certifications with objective tests that can be administered at conveniently located kiosks, at times that fit well with employees’ other responsibilities.

Switching from paper to the Apple iPad for delivery of observational assessments makes it possible for field trainers to watch a certification candidate demonstrate the use of a sophisticated medical device and rate his or her performance. Results are submitted  electronically for centralized processing. The observer’s comments are included along with their ratings. Field trainers no longer need to fax completed surveys to headquarters for scanning them into  people’s files.

Medical technicians who use vitro diagnostic systems must stay up to date with hematology, chemistry, hemostasis and other subjects related to screening blood samples – and they need t provide proof of their successful participation in continuing education programs. Data can be captured from post-training assessments to automatically generate certificates of participation in PDF format. This eliminates the need for printing out certificates and mailing them to each person.

Completing insurance forms is complex for anyone –and even more so for dermatologists with small practices who choose to handle this in-house. There are 800 different codes to master, and inaccurate coding can delay or negate payment.  Online certifications for dermatology coders have eliminated the costly, time consuming process of printing out printing tests, sending them to candidates and manually grading each one. Candidates can now log in to a test from home, work or elsewhere.

Organizations choose Questionmark for certification programmes for many reasons, including our collaborative browser-based Questionmark Live authoring system, multi-lingual test delivery capability, open architecture that allows easy linkage to scheduling software and our robust, secure and scalable delivery engine.

Another key reason for many is our blended delivery capability – where you author assessments once but then can deliver them on multiple devices, as shown in this diagram:

Questionmark Blended Delivery, author once and deliver to any device in context

For example, you can author an assessment and deliver it:

  • At a professional “bricks and mortar” test centre
  • At an office or training centre, with a manager or trainer overseeing
  • At a mobile test centre, for example one set up at a conference or trade show
  • At home with or without a secure browser or remote video monitoring
  • On mobile devices such as smartphones and tablets
  • On paper, with results scanned for centralized processing


Five Steps to Better Test Design and Delivery

Doug Peterson

Posted by Joan Phaup

I’ve been enjoying a series of posts in this blog my colleague Doug Peterson about Test Design and Delivery – so much so that I suggested he elaborate on this theme during a presentation at the Questionmark 2013 Users Conference in Baltimore March 3 – 6 – and he said, “Yes!”

Doug will be presenting on some other topics, too, but during a recent conversation with him I asked if he could tell me a little more about this particular session, which will focus on five processes:

1. Plan: Establish your test’s reliability and validity, and identify content areas to be covered
2. Create: Write items that increase the cognitive load, avoid bias and measure what’s important
3. Build: Pull items together into a test form, develop clear instructions and set passing scores
4. Deliver: Protect test content, control item exposure, protect test content and discourage cheating
5. Evaluate: Use item-, topic-, and test-level data to assess reliability and improve quality

Could you talk about your own background as a test author?

It mainly stems from what I was doing in the 3 or 4 years before I joined Questionmark. My group was responsible for training call center employees, and that included writing and administering lots of tests. Before my group took that over, all the tests were paper-and-pencil and had to be graded by the instructors. And of course you know that instructors over a period of time tend to bond with their students and tend to lose their objectivity.

It was clear that subjective testing was not good! We were introduced to Questionmark and we said, “Let’s automate these tests and make sure they are objective and fair.” That’s when I really got heavy-duty into testing. Over the course of those few years I attended several Questionmark conferences and went to a number sessions on item analysis, test analysis, setting cut scores, and so forth. I tried to understand all those kinds of things so that we could run statistical reports on our own content and sure our tests were valid and reliable.

Those years were very full of testing, and I learned a great deal about item and test writing, secure delivery and analyzing test results. I applied everything I learned to our call center training tests, and the customer satisfaction numbers began to rise. Why? Because our tests were working! The tests were valid and reliable, and because of that, they were weeding out the people who truly were not qualified for the job. Our stakeholders were very pleased because they had confidence that our tests were only passing people who were qualified to work in the call centers.

What do you think are the most challenging aspects of test design and delivery?

That’s hard to answer, because there are so many important things to think about! At the end of the day, the main thing is that the assessment is fair to both the test taker and the stakeholder, That idea encompasses many, many things. For the stakeholder, it requires having a valid, reliable assessment that uses solid methodology. For the participant, it boils down to well-written items. That sounds pretty simple, but it actually requires careful attention to detail.

How will you be addressing these challenges during your presentation at the Users Conference?

We’re going take a look at everything I’ve been working on in the blog series. What does reliable mean? What does valid mean? How can we appropriately plan an assessment and tie it back to the job or subject matter we’re testing for? We will also incorporate a lot of ideas about item writing. Throughout the session, we will be looking at fairness to the stakeholder and fairness to the participant and breaking those principles down into several components.

Who would benefit from attending this session?

Anyone who has anything to do with creating and delivering assessments: item writers, assessment assemblers, administrators. It’s good for people in these different roles to understand the entire test development and delivery process, so they appreciate their co-workers’ concerns. I see this session as suitable for people who are just beginning their work with Questionmark as well as those at the intermediate level. I’m looking forward to sharing so much of what I learned when I was so closely involved in a testing program myself.


There’s a lot to learn at this conference! Check out the agenda – and save $100 if you register by January 18th, 2013

Gear up for mobile assessment delivery with this webinar

Posted by Joan Phaup

Mobile devices such as smartphones and tablets offer a convenient and inexpensive way to deliver surveys and quizzes to learners anywhere, anytime.

If you would like to learn more about this cost-effective and flexible way to provide assessments for people on the move, join us for a free, one-hour Web seminar at 1 p.m. Eastern Time on Thursday, December 13: Creating Assessments for Mobile Delivery.

Find out how mobile delivery can be used on its own or as part of a blended delivery strategy to give learners retrieval practice, gather their opinions or test their knowledge and skills. And see how a single assessment can be delivered to many different types of devices, with all the results processed centrally.

This webinar offer answers to these and many other questions:

  • How can you create assessments that work well on small screens?
  • What are the most appropriate uses for such assessments?
  • What should you take into consideration when creating assessments for mobile delivery?
  • How QR codes impact assessment response rates?

Click here for complete webinar details and online registration.



Content protection and secure delivery: Test Design and Delivery Part 9

Posted By Doug Peterson

Writing good items and putting together valid and reliable assessments can take a lot of time and cost a lot of money. Part of an assessment’s reliability and validity is based on the test-taker not knowing the items ahead of time. For these reasons, it is critical that item exposure be controlled.

This starts during the development process by requiring everyone who develops items or assessments to sign a confidentiality agreement. Developers’ computers should, at the very least, be password-protected, and you should consider data encryption as well.

Care must be taken to prevent question theft once an assessment is assembled and delivered. Do not allow overly generous time limits, which would provide time for a test-taker to go back through the assessment and memorize questions. If your assessment is delivered electronically, consider not allowing backward movement through the test. Be very careful about allowing the use of a “scribble sheet”, as someone might try to write down questions and sneak them out of the test center: be sure to number all scribble sheets and collect them at the end of the assessment.

Computer-based testing makes it very easy to utilize random item selection when the assessment is delivered. While this does mean having to develop more items, it cuts down the number of times any one item is delivered and helps to reduce cheating by presenting different questions in a different order to teach test-taker.

It is critical to track the number of times an item has been delivered. After a certain number of deliveries, you will want to retire an item and replace it with a new item. The main factor that impacts how many times an item should be exposed is whether the assessment is high-stakes or low-stakes. Items on a high-stakes exam should have a lower maximum number of exposures, but items on a low-stakes exam can have a higher number of exposures.

As long as there have been tests, there have been test-takers who try to cheat. Make sure that you authenticate each examinee to ensure that the person who is supposed to be taking the exam is, in fact, the person taking the exam. Testing centers typically prohibit talking, using notes, and using cell phones during tests. Maintain a minimum amount of space between test-takers, or use carrels to physically separate them.

Test administrators should walk around the room during the test. Unauthorized personnel should not be permitted to enter the room during the test, and the administrator should not leave the room for any reason without first bringing in another administrator.

Computer-based testing presents its own set of security challenges, especially when testing is permitted outside of a secure testing center (e.g., in the test-taker’s home). Questionmark offers the Questionmark Secure client, which locks  down test-takers’ machines and doesn’t allow them to copy questions or switch tasks.

Computer-based testing security can/should also include some form of identification and password verification. Additionally, in the last few years, technology has become available that allows for the remote monitoring of test-takers using built-in laptop/tablet cameras or small desktop devices.

Click here for links to a complete listing of posts in this series.

Do certification exams give business benefit?

Posted by John Kleeman

Increasing numbers of technology vendors run certification programmes to help customers, employees and partners demonstrate competence in using or advising on the vendor’s technology. This is common in IT, in medical equipment, in the automotive industry and in many other high-tech industries.

Certification is an area where all stakeholders seem to be “winners”:

  • Vendors who set up certification programmes gain by being able to define the skill sets and knowledge that people deploying their technology need and encouraging stakeholders to develop the knowledge and skills and so deploy the technology more successfully for customers.
  • Participants benefit from certification as a way to learn and develop skills and demonstrate their competence, and it often helps in their career path.
  • Customers and users of the technology benefit from more effective deployment by being able to ensure the skills of experts deploying the technology and being more likely to get a successful implementation.
  • Employers of test-takers gain from their employees being more capable.

But how real is the benefit? How can you know if a well-designed and well-implemented certification programme will lead to improved performance?

There is some powerful evidence about this from an IDC study a few years back as reported on by Network World. This study looked at the benefit of certifications within IT network administration – surveying more than 1,000 IT managers. You can see some of the results in the chart below. For instance on average, unscheduled network downtime was about 20% lower at organizations with more certified IT staff.

Impacct of certifications on network administration

This study related to one particular field of IT, but it seems likely that in any technical field, providing you follow good practice in developing your certification programme, similar results should apply. Therefore certification is likely to provide material business benefits.

Questionmark is certainly seeing a lot of interest from customers looking to deliver certifications online, and I’ll talk about that in a follow-up post.