Getting Results — The Questionmark Blog : Discussing assessment trends and best practices, tech tips, case studies and more.

Q&A: Sue Martin and John Kleeman discuss steps to building a certification program

Posted by Zainab Fayaz

Certification programs are a vital way of recognizing knowledge, skills and professional expertise, but, during a time of digital transformation, how do you build a program that is sustainable and adaptable to the evolving needs of your organization, stakeholders and the market?

Questionmark Founder and Executive Director, John Kleeman and Sue Martin, certification expert and Business Transformation Consultant will present the upcoming webinar Building a Certification Program in 10 easy stepsat 4pm (GMT) / 11am Eastern / 5pm CET on 12 February 2019. However, before then, we wanted to find out a little bit more about what they’ll be covering during the session, so we’ve interviewed the two experts to gain insight!

Tell us a bit about what you’ll be covering during the webinar:

Sue: During the webinar, we’ll be covering a range of things; from the conceptual steps of building a certification program to the many projects that have evolved from these and the importance of outlining key steps from the very beginning of the process for creating a comprehensive and cohesive certification program.

We will also talk about the value certification program, can add to an organization, not only in the short-haul but also for many years to come. It is important to remember, “why” and “what” you are trying to achieve, and this webinar will provide detail on how the alignment of strategic goals and communication with stakeholders contributes to the success of an adaptable certification program.

John: We’ll be discussing a range of things during the webinar, but here are the ten easy steps that we’ll be describing:

  1. Business goals
  2. Scope
  3. Security
  4. Vendor evaluation
  5. Blueprint and test design
  6. Test development
  7. Pilot
  8. Communications
  9. Delivery
  10. Reporting and monitoring

What influenced the selection of these 10-steps you have identified in building a certification program?

John:  Sue and I sat down to plan the webinar when we were together at the OEB conference in Berlin in December. Although we wanted to cover a bit some of the obvious things like test design and development, we wanted to make sure people think first about the preparation and planning, for example getting organization buy-in and working out how to market and communicate the program to stakeholders. So we’ll be focusing on what you need to do to make a successful program, and that will drive everything you do

Although you’ll be covering the key steps for building a certification program during the webinar, can you advise on three key steps you find to be the most important during the process:

Sue:
1. Planning:
The emphasis of the program’s work should be at the start, in the planning phase – especially in order to build a flexible program which will adapt to the needs of your audience and stakeholders as their needs change over time. In all of the individual project components, whether it be test creation, vendor evaluation or communications rollout, for example, design and plan for the end goal. For example, when it comes to creating an exam, you plan for it right at the start of the project – you hit the ground running! It is not all about item writing, but also the development of the project from the beginning and if you don’t plan; this can lead to the lack of validity in the exam program and inconsistency over time

2. Practical tips and tricks for approaching various elements of your program development: It is important to set out the target audience; identify their learning journey and how they learn – in knowing this, can you go forward and build a certification program that can become integrated and aligns with the learning process

3. Scope: This is very important; setting the scope is a priority. Of course, in the greater scheme of things; you’ll have a mission statement, which provides you with a strategic vision, but when it comes to the finer detail and knowing what countries to enter, the pricing structure or knowing whether to offer remote proctoring; always keep in mind three things: the value contribution, the stakeholders and ask yourselves “yes, but why?”; as this will help align with organizational objectives.

What can attendees take away from the webinar you’ll present?  

Sue: Those attending will learn the value and importance of planning and questioning everything from the start of the process. We’ll share advice on the importance of having a value statement for every part of the process and making sure you know that a certification program is what you are looking for. By attending you can walk away with knowing the operational and strategic steps you must go through in order to build a program that is sustainable; think of it as a checklist!

John: If you’re starting a new certification program, I think this webinar will help guide you and help you create it more easily and more effectively. And if you already have a certification program and want to improve it, you’ll probably be doing a lot of what we suggest already but I hope they’ll be something for everyone to take away and learn.

Want to know more?

If you’re interested in learning more about the steps to building a certification program that meets the needs of your organization and stakeholders; then join John and Sue deliver a webinar session dedicated to Building a Certification Program in 10 easy steps on 12 February 2019 at 4pm (GMT) / 11am Eastern / 5pm CET.

You can register here.

A little bit more about our two experts:

John Kleeman is Executive Director and Founder of Questionmark. He has a first-class degree from Trinity College, Cambridge, and is a Chartered Engineer and a Certified Information Privacy Professional/Europe (CIPP/E). John wrote the first version of the Questionmark assessment software system and then founded Questionmark in 1988 to market, develop and support it. John has been heavily involved in assessment software for 30 years and has also participated in several standards initiatives including IMS QTI, ISO 23988 and ISO 10667. John was recently elected to the Association of Test Publishers (ATP) Board of Directors.

Sue Martin is a trusted advisor to companies and institutions across Europe in the area of workforce credentialing, learning strategies and certification. Her career prior to consulting included a role as Senior Global Certification Director for SAP and several regional and global management roles in the testing industry. She has also held several positions within industry institutions, such as the Chair of the European Association of Test Publishers and is currently a member of the Learning & Development Committee at BCS (British Computer Society).

 

Q&A: Microlearning and the Role of Measurement in Learning and Development at Progressive

 

Posted by Kristin Bernor

Chris Gilbert is a Senior Instructional Designer for Progressive Insurance, one of the largest providers of insurance in the United States. During his case study presentation at the 2019 Questionmark Conference in San Diego taking place from February 26 – March 1, he will talk about Using Questionmark to Build Microlearning for Photo Estimators. Progressive photo estimators use videos and photographs to identify damage and write estimates for necessary repairs.

This session will explore Progressive’s use of microlearning modules and the process they use to develop them.

I asked him recently about their case study:

Tell us about Progressive and your use of assessments:

At Progressive, we seek to make informed, data-driven decisions. We also strive to prepare and develop our people through effective, targeted learning solutions. In our learning orgs, assessments are one mechanism for gathering data we use to make a variety of decisions, including:

  • Identifying aspects and features of learning experiences that resonate within our target audiences so we can implement them in more of our deliverables
  • Pinpointing opportunities to improve learning experiences for our target audiences

Over the past few years, we’ve had a renewed focus on the importance of learning measurement and have established and implemented standards and tools for performing Level 1 and Level 2 measurement across and within all of our learning organizations. We’re currently working on Level 3 measurement to be able to measure and communicate the on-the-job impact of our learning experiences more consistently.

What do you mean by microlearning and why is it important to Progressive?

Microlearning is skill-based learning delivered in small “bite-sized” pieces. Microlearning can be developed in a variety of formats including videos, games, scenarios, and several others. Depending on the situation and needs of the organization and learners, microlearning can be delivered standalone, or as a supplement other learning experiences like in-person or virtual classroom courses.

Progressive is interested in adding microlearning into our learning deliverable portfolio for a variety of reasons, including:

  • Faster development times can improve our ability to deliver just-in-time learning solutions at the speed of modern business and change
  • Tightly-focused, skill-based topics and practice directly support on-the-job application
  • Today’s corporate learners seek quick-hit learning that gives them practical tools they need to succeed
  • Delivered in conjunction with other learning experiences, microlearning can help learners overcome the forgetting curve

What role does Questionmark play in ensuring that microlearning is successful?

One of the primary reasons, we decided to use Questionmark for our microlearning pilot project is that the data the system captures and it’s reporting capabilities allow us to provide the business with insights into several aspects of the learners’ performance in the modules. In turn, these insights will help the business make informed decisions.

What else about your session would you like to share?

Besides sharing the story of our first foray into microlearning, I’m planning to discuss some of the learnings we had related to question-type capabilities that we hadn’t previously explored.

Who would benefit most from attending this session and why?

a. Anyone interested in using Questionmark beyond its traditional use because the way we’re using it is a bit unconventional

b. Anyone interested in adding microlearning to their learning deliverable portfolio because Questionmark may provide a way for them to develop, deliver, and report the results

c. Anyone interested in extending the functionality of Questionmark question types to meet a business need because I’ll dig into some of the challenges, realizations, and learnings I experienced from having to extend a few of the question types in the pilot project

What are you especially looking forward to at this year’s Questionmark conference?

Meeting and networking with other Questionmark users, especially those who are passionate about the role of measurement in learning and development, and gaining more insight into how others are using system features and functionality in their organizations

Thank you Chris for taking time out of your busy schedule to discuss your session with us!

***

If you have not already done so, you still have a chance to attend this important learning event. Click here to register.

What time limit is fair to set for an online test or exam?

John KleemanPosted by John KleemanPicture of a sand timer

How do you know what time limit to set for a test or exam? I’m presenting a webinar on December 18th on some tips on how you can improve your tests and exams (it’s free of charge, register here) and this is one of the subjects I’ll be covering. In the meantime, this blog gives some good practice on setting a time limit.

Power tests

The first thing to identify is what the test is seeking to measure, and whether this has a speed element. Most tests are “power” tests in that they seek to measure someone’s knowledge or skill, not how fast it can be demonstrated. In a power test, you could set no time limit, but for practical purposes, it’s usual to set a time limit. This should allow most people to have enough time to complete answering the questions.

The best way to set a time limit is to pilot the test and measure how long pilot participants take to answer questions and use this to set an appropriate time period. If you have an established testing program, you may have organizational guidelines on time limits, for example you might allow a certain number of seconds or minutes per question; but even if you have such guidelines, you must still check that they are reasonable for each test.

Speed tests

Sometimes, speed is an important part of what you are trying to measure, and you need to measure that someone not only can demonstrate knowledge or skill but can also do so quickly. In a speed test, failure to be able to answer quickly may mean that the participant does not meet the requirements for what is being measured.

For example, in a compliance test for bank personnel to check their knowledge of anti-bribery and corruption laws, speed is probably not part of what is being measured. It will be rare in practice for people to encounter real-life issues involving bribery and very reasonable for them to think and consider before answering. But if you are testing a medical professional’s ability to react to a critical symptom in a trauma patient and make a decision on a possible intervention, rapid response is likely part of the requirement.

When speed is part of the requirements of what is being measured, the time limit for the test should be influenced by the performance requirements of the job or skill being measured.

Monitoring time limits

For all tests, it is important to review the actual time taken by participants to ensure that the time limit remains appropriate. You should regularly check the proportion of participants who answer all the questions in the test and those who skip or miss out some questions. In a speed test, it is likely that many participants will not finish the test. But if many participants are failing to complete a power test, then this should be investigated and may mean that the time limit is too short and needs extending.

If the time limit for a power test is too short, then essentially it becomes a speed test and is measuring how fast participants can demonstrate their skills. As such, if this is not part of the purpose of the test, it will impact the validity of the test results and it’s likely that the test will mis-classify people and so be unfair.

A particular point of concern is when you are using computerized tests to test people who are not proficient computer users. They will inevitably be slower than proficient computer users, and unless your test seeks to measure computer proficiency, you need to allow such people enough time.

What about people who need extra time?

It’s common to give extra time as accommodation for certain kinds of special needs. Extra time is also sometimes given for linguistic reasons e.g. taking an assessment in second language. Make sure that your assessment system lets you override the time limit in such cases. Ideally base the extra time in such cases on piloting, not just a fixed extra percentage.

Screenshot showing a setting where it is possible to exclude material from the assessment time limitWhen should a time limit start?

My last tip is that the time limit should only start when the questions begin. If you are presenting any of these:

  • Introductory material or explanation
  • Practice questions
  • An honor code to commit to staying honest and not cheating
  • Demographic questions

The time limit should start after these are done. If you are using Questionmark software, you can make this happen by excluding the question block from the assessment time limit.

 

If you are interested in more tips on improving your tests and exams, register to attend our free webinar on December 18th:  10 Quick Tips to Improve your Tests and Exams.

What is the Single Best Way to Improve Assessment Security?

John KleemanPosted by John Kleeman

Three intersecting circles, one showing Confidentiality, one showing Availability and one showing IntegrityAssessment results matter. Society relies on certifications and qualifications granted to those who pass exams. Organizations take important decisions about people based on test scores. And individuals work hard to learn skills and knowledge they can demonstrate in tests and exams. But in order to be able to trust assessment results, the assessment process needs to be secure.

Security is usefully broken down into three aspects: confidentiality, integrity and availability.

  • Confidentiality for assessments includes that questions are kept secure and that results are available only to those who should see them.
  • Integrity for assessments includes that that the process is fair and robust, that identify of the test-taker is confirmed and that cheating does not take place.
  • Availability includes that assessments can be taken when needed and that results are stored safely for the long term.

A failure of security, particularly one of confidentiality or integrity reduces the usefulness and trustworthiness of test results. A confidentiality failure might mean that results are meaningless as some test-takers knew questions in advance. An integrity failure means that some results might not be genuine.

So how do you approach making an assessment program secure? The best way to think about this is in terms of risk. Risk assessment is at the heart of all successful security systems and central to the widely respected ISO 27001 and NIST 800-53 security standards. In order to focus resources to make an assessment program secure and to reduce cheating, you need to enumerate and quantify the risks and identify probability (how likely they are to happen) and impact (how serious it is if they do). You then allocate mitigation effort at the ones with higher probability and impact. This is shown illustratively in the diagram – the most important risks to deal with are those that have high probability and high impact.

Four quadrants showing high probability, high impact in red and Low probability, low impact in green. With yellow squares for high probability, low impact and low probability, high impact

One reason why risk assessment is sensible is that it focuses effort on issues that matter. For example, the respected Verizon data breach investigations report for 2017 reported that 81% of hacking-related breaches involved weak or stolen passwords. For most assessment programs, it will make sense to put in place measures like strong passwords and training on good password practice for assessment administrators and authors to help mitigate this risk.

There is no “one size fits all approach”. Some risks will differ between assessment programs. To give a simple example, some organizations are concerned  about people having reference materials or “cheat sheets” to look up answers in and this can be an important risk to mitigate against; whereas in other programs, exams are open book and this is not a concern. In some programs, identity fraud (where someone pretends to be someone else to take the exam for them) is a big concern; in others the nature of the proctoring or the community makes this much less likely.

If you’re interested in learning more about the risk approach to assessment security, I’m presenting a webinar “9 Risks to Test Security (and what to do about them)” on 28th November which:

  • Explains the risk approach to assessment security.
  • Details nine key risks to assessment security from authoring through delivery and into reporting.
  • Gives some real examples of the threats for each risk.
  • Suggests some mitigations and measures to consider to improve security.

You can see more details on the webinar and register here.

Assessment security matters because it impacts the quality and trustworthiness of assessment results. If you are not already doing it, starting a risk-based approach to analyzing risks to your security is the single best way to improve assessment security.

How to Navigate Assessments through the GDPR Automated Decision-Making Rules

John KleemanPosted by John Kleeman

The GDPR has got a lot of publicity for its onerous consent requirements, large fines and the need to inform of data breaches. But there are other aspects of GDPR which have implications for assessment users. To protect human rights, the GDPR imposes restrictions on letting machines make decisions about people, and these limitations can apply when using computerized assessments. Here is how one of the recitals to the GDPR describes the principle:

“The data subject should have the right not to be subject to a decision … evaluating personal aspects relating to him or her which is based solely on automated processing and which produces legal effects concerning him or her or similarly significantly affects him or her, such as automatic refusal of an online credit application or e-recruiting practices without any human intervention.”

In some cases, it is actually illegal in the European Union to use a computerized test or exam to make a significant decision about a person. In other cases, it is permissible but you need to put in place specific measures.  The assessment industry has always been very careful about reliability, validity and fairness of tests and exams, so these measures are navigable, but you need to follow the rules. The diagram below shows what is allowed, with or without protection measures in place, and what is forbidden.

Flowchart describing rules on automated decision-making in the GDPR

 

 

When you are free from restriction

For many assessments, the GDPR rules will not impose any prohibitions, as shown by the green “Allowed” box in the diagram:

  • If you are only making minor decisions from an assessment, you do not need to worry.  For example, if you are delivering e-learning, and you decide which path to go next depending on an assessment, that is unlikely to significantly impact the assessment participant.  But if the assessment impacts significant things, like jobs, promotions or access to education, or has a legal effect, the restrictions will apply.
  • Even if decisions made do have legal or significant effects, the GDPR only restricts solely automated decision-making. If humans are genuinely part of the decision process, for example with the ability to change the decision, this is not solely automated decision-making. This doesn’t mean that an assessment is okay if humans wrote the questions or set the pass score; it means that humans must review the results before making a decision about a person based on the test. For example, if a recruitment test screens someone automatically out of a job application process without a person intervening, the GDPR consider this to be solely automated decision-making. But if an employee fails a compliance test, and this is referred to a person who reviews the test results and other information and genuinely decides the action to take, that is not solely automated decision making.

What to do if the restrictions apply

If the GDPR restrictions do apply, you have to go through some logic as shown in the diagram to see if you are permitted to do this at all. If you do not fall into the permitted cases, it will be illegal to make the decision according to the GDPR (the red boxes). In other cases, it is permitted to use automated decision-making, but you have to put measures in place (the yellow boxes). Here are some of the key measures a data controller (usually the assessment sponsor) may take if the yellow boxes apply, for example when using assessments in screening candidates for recruiting:

  1. Provide a route where test takers can appeal the assessment result and the decision and have a human review;
  2. Inform test takers that you are using automated decision making and what the consequences for them will be;
  3. Provide meaningful information about the logic involved. I suggest this might include publishing an explanation of how questions are created and reviewed, how the scoring works and in a pass/fail test, how the pass score is arrived at fairly;
  4. Have mechanisms in place to ensure the ongoing quality and fairness of the test. The regulators aren’t precise about what you need to do, but one logically important thing would be to ensure that the question and test authoring process results in a demonstrably valid and reliable test. And to maintain validity and reliability, it’s important to conduct regular item analysis and other reviews to ensure quality is maintained.
  5. Perform and document a Data Protection Impact Assessment (DPIA) to check that test taker’s rights and interests are being respected, if the assessment will involve a systematic and extensive evaluation of personal aspects relating to the test taker or otherwise gives a high risk to rights.  Questionmark has produced a template for DPIAs which might help here – see www.questionmark.com/go/eu-od-dpiatemplate.

Although these measures might appear threatening on first sight, in fact they could be helpful for the quality of assessments. As I describe in my blog post What is the best way to reduce cheating?, providing information to test-takers about how the test is created and scored and why this is fair, can help reduce cheating by making the test-taker less likely to rationalize  that cheating is fair. And it is generally  good practice to use an assessment as one piece of data along with other criteria to make a decision about someone. The increased visibility and transparency of the assessment process by following the requirements above could also encourage better practice in assessment, and so more reliable, valid and trustable assessments for all.

If you want to find out more about the rules, there is guidance available from the European Data Protection Board and from the UK Information Commissioner. Questionmark customers who have questions in this area are also welcome to contact me. You might also like to read Questionmark’s white paper “Responsibilities of a Data Controller When Assessing Knowledge, Skills and Abilities” which you can download here.

This blog post includes my personal views only and is based on guidance currently available on the GDPR. This is a fluid area that is likely to develop over time, including through publication of additional regulator guidance and court decisions. This blog does not constitute legal advice.

Ten Tips to Translate Tests Thoughtfully

John KleemanPosted by John Kleeman

Tests and exams are used for serious purposes and have significant impact on people’s lives. If they are translated wrongly, it can result in distress. As a topical illustration, poor translation of an important medical admissions test in India was the subject of a major law case ruled on by the Indian Supreme Court last week.

Because language and cultures vary, fairly translating tests and exams is hard. I recently attended a seminar organized by the OECD on translating large scale assessments which gave me a lot of insight into the test translation process.  If you are interested  in the OECD seminar, Steve Dept of Questionmark partner cApStAn has written a blog here, and the seminar presentations are available on the OECD website.

Here are some tips from what I’ve learned at the seminar and elsewhere on good practice in translating tests and exams.

  1. Put together a capable translation management team. A team approach works well when translating tests. For example a subject matter expert, a linguist/translator, a business person and a testing expert would work well together as a review and management committee.
  2. Think through the purpose of your translation. Experts say that achieving perfect equivalence of a test in two languages is close to impossible, so you need to define your goals. For example, are you seeking to adapt the test to measure the same thing or are you looking for a literal translation? The former may be more realistic especially if your test includes some culturally specific examples or context.  Usually what you will be looking for is that the test in two languages is comparable in that a pass score in the test in either language means a similar thing for competence.
  3. Define a glossary for your project. If your test is on a specialist or technical subject, it will have some words specific to the content area. You can save time and increase the quality of the translation if you identify the expected translation of these words in advance. This will guide the translating team and ensure that test takers see consistent vocabulary.
  4. Use a competent translator (or translation company). A translator must be native in the target language but also needs current cultural knowledge, ideally from living in the target locale. A translator who is not native to the language will not be effective, and a translator who does not have knowledge of the culture may miss some references in question content  (e.g. local names or slang). An ideal translator will also have subject matter knowledge and assessment knowledge.
  5. Diagram showing export into XLIFF XML and then re-importExport to allow a translator to use their own tools. Translators have many automated tools available to them including translation memories, glossaries and automated checking systems. For simple translation, you can translate interactively within an assessment system, but you will get more professional results if you export from your assessment management system, allow the translator to translate in their system, and then re-import (as shown in the diagram).
  6. Put in place a verification procedure. Translators are human and make mistakes, questions can also rely on context or knowledge that a translator may not have. A verification process will involve manual review by stakeholders looking at things like accuracy, style, country issues, culture, no clues given in choices, right choice not obviously longer than other choices and different translation word choices used in stem/choices.
  7. Also review by piloting and looking at item difficulty. Linguistic review is helpful but you should also look at item performance in practice. The difficulty of a translated item will vary slightly between languages. Generally small errors will be up and down and roughly cancel out. You want to catch the big errors, where ambiguity or mis-translation makes a material difference to test accuracy. You can catch some of these by running a small pilot to 50 (or even 25) participants and comparing the p-value (item difficulty or proportion who get right) in the languages. This can flag questions with significant differences in difficulty; such questions need review as they may well be badly translated.
  8. Consider using bilingual reviewers. If you have access to bilingual people (who speak the target and source language), it can be worth asking them to look at both versions of the questions and comment. This shouldn’t be your only verification procedure but can be very helpful and spot issues.
  9. Update translations as questions change. In any real world test, questions in your item bank get updated over time, and that means you need to update the translations and keep track of which ones have been updated in which languages. It can be helpful  to use a translation management system, for example the one included within Questionmark OnDemand to help you manage this process, as it’s challenging and error-prone to manage manually.
  10. Read community guidelines. The International Test Commission have produced well-regarded guidelines on adapting/ translating tests – you can access them here. The OECD PISA guidelines, although specific to the international PISA tests, have  good practice applicable to other programs. I personally like the heading to one of the sections in the PISA guidance: “Keep in mind that some respondents will misunderstand anything that can be misunderstood”!

I hope you found this post interesting – all suggestions are personal and not validated by the OECD or others. If you did find it interesting, you may also want to read my earlier blog post: Twelve tips to make questions translation ready.

To learn more about Questionmark OnDemand and Questionmark’s translation management system, see here or request a demo.