What time limit is fair to set for an online test or exam?

John KleemanPosted by John KleemanPicture of a sand timer

How do you know what time limit to set for a test or exam? I’m presenting a webinar on December 18th on some tips on how you can improve your tests and exams (it’s free of charge, register here) and this is one of the subjects I’ll be covering. In the meantime, this blog gives some good practice on setting a time limit.

Power tests

The first thing to identify is what the test is seeking to measure, and whether this has a speed element. Most tests are “power” tests in that they seek to measure someone’s knowledge or skill, not how fast it can be demonstrated. In a power test, you could set no time limit, but for practical purposes, it’s usual to set a time limit. This should allow most people to have enough time to complete answering the questions.

The best way to set a time limit is to pilot the test and measure how long pilot participants take to answer questions and use this to set an appropriate time period. If you have an established testing program, you may have organizational guidelines on time limits, for example you might allow a certain number of seconds or minutes per question; but even if you have such guidelines, you must still check that they are reasonable for each test.

Speed tests

Sometimes, speed is an important part of what you are trying to measure, and you need to measure that someone not only can demonstrate knowledge or skill but can also do so quickly. In a speed test, failure to be able to answer quickly may mean that the participant does not meet the requirements for what is being measured.

For example, in a compliance test for bank personnel to check their knowledge of anti-bribery and corruption laws, speed is probably not part of what is being measured. It will be rare in practice for people to encounter real-life issues involving bribery and very reasonable for them to think and consider before answering. But if you are testing a medical professional’s ability to react to a critical symptom in a trauma patient and make a decision on a possible intervention, rapid response is likely part of the requirement.

When speed is part of the requirements of what is being measured, the time limit for the test should be influenced by the performance requirements of the job or skill being measured.

Monitoring time limits

For all tests, it is important to review the actual time taken by participants to ensure that the time limit remains appropriate. You should regularly check the proportion of participants who answer all the questions in the test and those who skip or miss out some questions. In a speed test, it is likely that many participants will not finish the test. But if many participants are failing to complete a power test, then this should be investigated and may mean that the time limit is too short and needs extending.

If the time limit for a power test is too short, then essentially it becomes a speed test and is measuring how fast participants can demonstrate their skills. As such, if this is not part of the purpose of the test, it will impact the validity of the test results and it’s likely that the test will mis-classify people and so be unfair.

A particular point of concern is when you are using computerized tests to test people who are not proficient computer users. They will inevitably be slower than proficient computer users, and unless your test seeks to measure computer proficiency, you need to allow such people enough time.

What about people who need extra time?

It’s common to give extra time as accommodation for certain kinds of special needs. Extra time is also sometimes given for linguistic reasons e.g. taking an assessment in second language. Make sure that your assessment system lets you override the time limit in such cases. Ideally base the extra time in such cases on piloting, not just a fixed extra percentage.

Screenshot showing a setting where it is possible to exclude material from the assessment time limitWhen should a time limit start?

My last tip is that the time limit should only start when the questions begin. If you are presenting any of these:

  • Introductory material or explanation
  • Practice questions
  • An honor code to commit to staying honest and not cheating
  • Demographic questions

The time limit should start after these are done. If you are using Questionmark software, you can make this happen by excluding the question block from the assessment time limit.

 

If you are interested in more tips on improving your tests and exams, register to attend our free webinar on December 18th:  10 Quick Tips to Improve your Tests and Exams.

How to Navigate Assessments through the GDPR Automated Decision-Making Rules

John KleemanPosted by John Kleeman

The GDPR has got a lot of publicity for its onerous consent requirements, large fines and the need to inform of data breaches. But there are other aspects of GDPR which have implications for assessment users. To protect human rights, the GDPR imposes restrictions on letting machines make decisions about people, and these limitations can apply when using computerized assessments. Here is how one of the recitals to the GDPR describes the principle:

“The data subject should have the right not to be subject to a decision … evaluating personal aspects relating to him or her which is based solely on automated processing and which produces legal effects concerning him or her or similarly significantly affects him or her, such as automatic refusal of an online credit application or e-recruiting practices without any human intervention.”

In some cases, it is actually illegal in the European Union to use a computerized test or exam to make a significant decision about a person. In other cases, it is permissible but you need to put in place specific measures.  The assessment industry has always been very careful about reliability, validity and fairness of tests and exams, so these measures are navigable, but you need to follow the rules. The diagram below shows what is allowed, with or without protection measures in place, and what is forbidden.

Flowchart describing rules on automated decision-making in the GDPR

 

 

When you are free from restriction

For many assessments, the GDPR rules will not impose any prohibitions, as shown by the green “Allowed” box in the diagram:

  • If you are only making minor decisions from an assessment, you do not need to worry.  For example, if you are delivering e-learning, and you decide which path to go next depending on an assessment, that is unlikely to significantly impact the assessment participant.  But if the assessment impacts significant things, like jobs, promotions or access to education, or has a legal effect, the restrictions will apply.
  • Even if decisions made do have legal or significant effects, the GDPR only restricts solely automated decision-making. If humans are genuinely part of the decision process, for example with the ability to change the decision, this is not solely automated decision-making. This doesn’t mean that an assessment is okay if humans wrote the questions or set the pass score; it means that humans must review the results before making a decision about a person based on the test. For example, if a recruitment test screens someone automatically out of a job application process without a person intervening, the GDPR consider this to be solely automated decision-making. But if an employee fails a compliance test, and this is referred to a person who reviews the test results and other information and genuinely decides the action to take, that is not solely automated decision making.

What to do if the restrictions apply

If the GDPR restrictions do apply, you have to go through some logic as shown in the diagram to see if you are permitted to do this at all. If you do not fall into the permitted cases, it will be illegal to make the decision according to the GDPR (the red boxes). In other cases, it is permitted to use automated decision-making, but you have to put measures in place (the yellow boxes). Here are some of the key measures a data controller (usually the assessment sponsor) may take if the yellow boxes apply, for example when using assessments in screening candidates for recruiting:

  1. Provide a route where test takers can appeal the assessment result and the decision and have a human review;
  2. Inform test takers that you are using automated decision making and what the consequences for them will be;
  3. Provide meaningful information about the logic involved. I suggest this might include publishing an explanation of how questions are created and reviewed, how the scoring works and in a pass/fail test, how the pass score is arrived at fairly;
  4. Have mechanisms in place to ensure the ongoing quality and fairness of the test. The regulators aren’t precise about what you need to do, but one logically important thing would be to ensure that the question and test authoring process results in a demonstrably valid and reliable test. And to maintain validity and reliability, it’s important to conduct regular item analysis and other reviews to ensure quality is maintained.
  5. Perform and document a Data Protection Impact Assessment (DPIA) to check that test taker’s rights and interests are being respected, if the assessment will involve a systematic and extensive evaluation of personal aspects relating to the test taker or otherwise gives a high risk to rights.  Questionmark has produced a template for DPIAs which might help here – see www.questionmark.com/go/eu-od-dpiatemplate.

Although these measures might appear threatening on first sight, in fact they could be helpful for the quality of assessments. As I describe in my blog post What is the best way to reduce cheating?, providing information to test-takers about how the test is created and scored and why this is fair, can help reduce cheating by making the test-taker less likely to rationalize  that cheating is fair. And it is generally  good practice to use an assessment as one piece of data along with other criteria to make a decision about someone. The increased visibility and transparency of the assessment process by following the requirements above could also encourage better practice in assessment, and so more reliable, valid and trustable assessments for all.

If you want to find out more about the rules, there is guidance available from the European Data Protection Board and from the UK Information Commissioner. Questionmark customers who have questions in this area are also welcome to contact me. You might also like to read Questionmark’s white paper “Responsibilities of a Data Controller When Assessing Knowledge, Skills and Abilities” which you can download here.

This blog post includes my personal views only and is based on guidance currently available on the GDPR. This is a fluid area that is likely to develop over time, including through publication of additional regulator guidance and court decisions. This blog does not constitute legal advice.

The Nineteen Responsibilities of an Assessment Data Controller under the GDPR

John KleemanPosted by John Kleeman

Back in 2014,  Questionmark produced a white paper covering what at the time was a fairly specialist subject – what assessment organizations needed to do to ensure compliance with European data protection law. With the GDPR in place in 2018, with its extra-territorial reach and potential of large fines, the issue of data protection law compliance is one that all assessment users need to consider seriously.

Data Controller with two Data Processors, one of which has a Sub-Processor

Myself, Questionmark Associate Legal Counsel Jamie Armstrong and Questionmark CEO Eric Shepherd have now rewritten the white paper to cover the GDPR and published it this week. The white paper is called  “Responsibilities of a Data Controller When Assessing Knowledge, Skills and Abilities”. I’m pleased to give you a summary in this blog article.

To remind you, a Data Controller is the organization responsible for making decisions about personal data, whereas a Data Processor is an organization who processes data on behalf of the Data Controller. As shown in the diagram, a Data Processor may have Sub-Processors. In the assessment context, examples of Data Controllers might be:

  • A company that tests its personnel for training or regulatory compliance purposes;
  • A university or college that tests its students;
  • An awarding body that gives certification exams.

Data Processors are typically companies like Questionmark that provide services to assessment sponsors. Data Processors have significant obligations under the GDPR, but the Data Controller has to take the lead.  The Nineteen Responsibilities of an Assessment Data Controller under the GDPR 1. Ensure you have a legitimate reason for processing personal data 2. Be transparent and provide full information to test-takers 3. Ensure that personal data held is accurate 4. Review and deal properly with any rectification requests 5. Respond to subject access requests 6. Respond to data portability requests 7. Delete personal data when it is no longer needed 8. Review and deal properly with any erasure requests 9. Put in place strong security measures 10. Use expert processors and contract with them wisely 11. Adopt privacy by design measures 12. Notify personal data breaches promptly 13. Consider whether you need to carry out a Data Protection Impact Assessment 14. Follow the rules if moving data out of Europe 15. If collecting “special” data, follow the particular rules carefully 16. Include meaningful human input as well as assessment results in making decisions 17. Respond to restriction and objection requests 18. Train your personnel effectively 19. Meet organisational requirementsBack in 2014, we considered there were typically 12 responsibilities for an assessment Data Controller. Our new white paper suggests there are now 19. The GDPR significantly expands the responsibilities Data Controllers have as well as makes it clearer what needs to be done and the likely penalties if it is not done.

The 25 page white paper:

  • Gives a summary of European data protection law
  • Describes what we consider to be the 19 responsibilities of a Data Controller (see diagram)
  • Gives Data Controllers a checklist of the key measures they need from a Data Processor to be able to meet these responsibilities
  • Shares how Questionmark helps meet the responsibilities
  • Comments on how the GDPR by pushing for accuracy of personal data might encourage more use of valid, reliable and trustworthy assessments and benefit us all

The white paper is useful reading for anyone who delivers tests and exams to people in Europe – whether using Questionmark technology or not. Although we hope it will be helpful, like all our blog articles and white papers, this article and the white paper are not a substitute for legal advice specific to your organization’s circumstances. You can see and download all our white papers at www.questionmark.com/learningresources and you can directly download this white paper here.

Six tips to increase reliability in competence tests and exams

Posted by John Kleeman

Reliability (how consistent an assessment is in measuring something) is a vital criterion on which to judge a test, exam or quiz. This blog post explains what reliability is, why it matters and gives a few tips on how to increase it when using competence tests and exams within regulatory compliance and other work settings

What is reliability?

Picture of a kitchen scaleAn assessment is reliable if it measures the same thing consistently and reproducibly.

If you were to deliver an assessment with high reliability to the same participant on two occasions, you would be very likely to reach the same conclusions about the participant’s knowledge or skills. A test with poor reliability might result in very different scores across the two instances.

It’s useful to think of a kitchen scale. If the scale is reliable, then when you put a bag of flour on the scale today and the same bag of flour on tomorrow, then it will show the same weight. But if the scale is not working properly and is not reliable, it could give you a different weight each time.

Why does reliability matter?

Just like a kitchen scale that doesn’t work, an unreliable assessment does not measure anything consistently and cannot be used for any trustable measure of competency.

As well as reliability, it’s also important that an assessment is valid, i.e. measures what it is supposed to. Continuing the kitchen scale metaphor, a scale might consistently show the wrong weight; in such a case, the scale is reliable but not valid. To learn more about validity, see my earlier post Six tips to increase content validity in competence tests and exams.

How can you increase the reliability of your assessments?

Here are six practical tips to help increase the reliability of your assessment:

  1. Use enough questions to assess competence. Although you need a sensible balance to avoid tests being too long, reliability increases with test length. In their excellent book, Criterion-Referenced Test Development, Shrock and Coscarelli suggest a rule of thumb is 4-6 questions per objective, with more for critical objectives. You can also get guidance from an earlier post on this blog How many questions do I need on my assessment?
  2.  Have a consistent environment for participants. For test results to be consistent, it’s important that the test environment is consistent – try to ensure that all participants have the same amount of time to take the test in and have a similar environment. For example, if some participants are taking the test in a hurry in a public and noisy place and others are taking it at leisure in their office, this could impact reliability.
  3. Ensure participants are familiar with the assessment user interface. If a participant is new to the user interface or the question types, then they may not show their true competence due to the unfamiliarity. It’s common to provide practice tests to participants to allow them to become familiar with the assessment user interface. This can also reduce test anxiety which also influences reliability.
  4. If using human raters, train them well. If you are using human raters, for example in grading essays or in observational assessments that check practical skills, make sure to define your scoring rules very clearly and as objectively as possible. Train your observers/raters, review their performance, give practice sessions and provide exemplars.
  5. Measure reliability. There are a number of ways of doing this, but the most common way is to calculate what is called “Cronbach’s Alpha” which measures internal consistency reliability (the higher it is, the better). It’s particularly useful if all questions on the assessment measure the same construct. You can easily calculate this for Questionmark assessments using our Test Analysis Report.
  6. Conduct regular item analysis to weed out ambiguous or poor performing questions. Item analysis is an automated way of flagging weak questions for review and improvement. If questions are developed through sound procedures and so well crafted and non-ambiguously worded they are more likely to discriminate well and so contribute to a reliable test. Running regular item analysis is the best way to identify poorly performing questions. If you want to learn more about item analysis, I recently gave a webinar on “Item Analysis for Beginners”, and you can access the recording of this here.

 

I hope this blog post reminds you why reliability matters and gives some ideas on how to improve reliability. There is lots more information on how to improve reliability and write better assessments on the Questionmark website – check out our resources at www.questionmark.com/learningresources.

Six predictions now the GDPR is in place

Posted by John Kleeman
So the European GDPR is in place now. Questionmark like most other companies has been working hard in the last two years to ensure we are compliant and that our customers in and outside Europe can be compliant with the GDPR. See our trust center or summary for information on Questionmark’s compliance.

Is it all done and dusted? My email inbox seems to have a few less promotional emails in it. But is this because of the holiday weekend or have companies really taken my name off their mailing lists? Here are six predictions for what we’ll see going forwards with the GDPR.

1. The May 25th 2018 date will matter much less going forwards than backwards

A picture of a dog with a Christmas hatCompanies have been rushing to meet the May 25th date, but GDPR and privacy is a destination not a journey. There is a famous slogan “a dog is for life not just for Christmas” encouraging people to look after their dog and not just buy it as a cute puppy. Similarly the GDPR is not just something you get compliant with and then ignore. You need to include privacy and compliance in your processes forever.

No one will care much whether you were compliant on May 25th 2018. But everyone will care whether you are meeting their privacy needs and following the law when they interact with you.

2. History will judge the GDPR as a watershed moment where privacy became more real

Nevertheless I do think that history will judge the GDPR as being a seminal moment for privacy. Back in the early 2000s, Microsoft popularized the concept of security by design and security by default when they delayed all their products for a year as they improved their security. Nowadays almost everyone builds security into their systems and makes it the default because you have to to survive.

Similarly the GDPR encourages us to think of privacy when we design products and to make privacy the default not an afterthought. For example, when we collect data, we should plan how long to keep it and how to erase it later. I suspect in ten years time, privacy by design will be as commonplace as security by design – and the GDPR will be the key reason it became popular.

3. Many other jurisdictions will adopt GDPR like laws

Although the GDPR is over-complex, it has some great concepts in it, that I’m sure other countries will adopt. It is appropriate that organizations have to take care about processing peoples’ data. It is appropriate that when you pass people’s data onto a third party, there should be safeguards. And if you breach that data, it is appropriate that you should have to be held accountable.

We can expect lawmakers in other countries to make GDPR-like laws.

4. Supply chain management will become more important

Diagram showing one data controller with two data processors. One data processor has two sub-processors and one data processor has one sub-processorUnder the GDPR, a Data Controller contracts with Data Processors and those Data Processors must disclose their Sub-processors (sub-contractors). There is positive encouragement to choose expert Data Processors and Sub-processors and there are consequences if processors fail their customers. This will encourage organizations to choose reputable suppliers and to review processors down the chain to make sure that everyone is following the rules. Choosing suppliers and Sub-processors that get themselves audited for security, e.g. under ISO 27001, is going to become more commonplace.

This will mean that some suppliers who do not have good enough processes in place for security, privacy and reliability will struggle to survive.

5. People will be the biggest cause of compliance failures

Organizations set up processes and procedures and put in place systems and technology to run their operations, but people are needed to design and run those processes and technology. Some GDPR compliance failures are going to be down to technology failures, but I predict the majority will be down to people. People will make mistakes or judgement errors and cause privacy and GDPR breaches.

If you are interested in this subject, Amanda Maguire of SAP and I gave a webinar last week entitled “GDPR is almost here – are your people ready?” which should shortly be available to view on the SAP website. The message we shared is that if you want to stay compliant with the GDPR, you need to check your people know what to do with personal data. Testing them regularly is a good way of checking their knowledge and understanding.

6. The GDPR and privacy concerns will encourage more accurate assessments

Last but not least, I think that the GDPR will encourage people to expect more accurate and trustworthy tests and exams. The GDPR requires that we pay attention to the accuracy of personal data; “every reasonable step must be taken to ensure that personal data that are inaccurate … are erased or rectified without delay”.

There is a strong argument this means that if someone creates a test or exam to measure competence, that the assessment should be accurate in what it claims to measure. So it needs to be authored using appropriate procedures to make it valid, reliable and trustworthy. If someone takes an assessment which is invalid or unfair, and fails it, they might reasonably argue that the results are not an accurate indication of their competence and so that personal data is inaccurate and needs correcting.

For some help on how you can make more accurate assessments, check out Questionmark white papers at www.questionmark.com/learningresources including “Assessment Results You Can Trust”.

 

 

Washington DC – OnDemand for Government Briefing Recap

Posted by Kristin Bernor

Last Thursday, May 17, Questionmark hosted a briefing in Washington, DC that powerfully presented the journey to delivering the Questionmark OnDemand for Government assessment management system to our government customers. We would like to thank industry experts that made this possible including speakers from the Department of State, FedRAMP, Microsoft and Schellman. In just 3 1/2 hours, they were able to present the comprehensive process of what goes into delivering the Questionmark OnDemand for Government system to market. This new government community cloud-based service dedicated to the needs of U.S. governmental and defense agencies is currently wrapping up the onsite portion of the audit. The auditors will be finalizing their testing and review culminating in an assessment report. The complete security package is expected to be available in July. Questionmark OnDemand for Government is designed to be compliant with FedRAMP and hosted in a FedRAMP certified U.S. data center.

Highlights from the briefing included presentations from:

  • Eric Shepherd, CEO of Questionmark, hosted the event.
  • Ted Stille, Department of State, discussed the agency’s motivations and experience as project sponsor for Questionmark OnDemand for Government.
  • Stacy Poll and David Hunt, Public Sector Business Development Manager and Information Security Officer of Questionmark respectively, presented a system overview including demonstration screens, migration paths and detailed next steps to plan for implementation.
  • Christina McGhee, Schellman audit team, spoke about the 3PAO role in the FedRAMP authorization process.
  • Zaree Singer and Laurie Southerton, FedRAMP PMO Support, explained the FedRAMP ATO approval process.
  • Ganesh Shenbagaraman, Microsoft, discussed Microsoft Azure’s government cloud service.

This unique opportunity to learn about the OnDemand for Government assessment management system, meet with peers and other customers, and hear from FedRAMP and our 3PAO themselves proved invaluable to attendees who rated the briefing a near 5 out of 5.

Please reach out to Stacy Poll at stacy.poll@questionmark.com for more information.