Will 2016 mark the end of test centers?

John Kleeman HeadshotPosted by John Kleeman

The Internet is causing disruptive change in many industries. When I was a child, we had a bookshelf full of encyclopaedias, nowadays my children can find out much more than was ever in these large books on Google or Wikipedia. This is just one of many disruptions the Internet has brought.

Building being demolishedCurrently many high-stakes exams are taken at “bricks and mortar” test centers. Participants need to travel to a test center to take the exam, often having to journey a considerable distance; tests need to be scheduled for a fixed slot in advance; and participants or their employer must pay a substantial fee for use of the test center. Increasingly, the Internet is allowing a new dynamic, with participants taking tests at their offices or homes, with a remote, online proctor ensuring integrity.

So will the Internet take another scalp? Will 2016 be the year that test centers for high stakes tests and exams start to disappear? Will there even be bulldozers demolishing test centers or re-purposing them for other tasks?

Interested in learning more about online proctoring? I will be presenting a session on Ensuring Exam Integrity with Online Proctoring at the Questionmark Conference 2016: Shaping the Future of Assessment in Miami, April 12-15. I look forward to seeing you there! Click here to register and learn more about this important learning event. 

It may be a while before the bulldozers move in, but as shown in my previous post Online or test center proctoring: Which is best?, there are many advantages of online proctoring. Questionmark exams can be delivered both at test centers and via online proctoring, but I am increasingly coming to believe that online proctoring is the future for high stakes exams.

The benefit to the participant of not having to travel to a test center and being able to take the exam in a flexible timeslot is immense. To illustrate this, here is a quote from an independent blog entry by a candidate describing his experience taking an SAP certification exam in 2015:

“I was able to take my test from my home office. This is such a big improvement because I did not have to drive to a test centre which in my case is far away and where I can only take the test on specific days in the year.”

The main reason that organizations use test centers is that it feels more secure. But as I suggested in Online or test center proctoring: Which is more secure?, there are arguments on both sides for security, and most organizations probably use test centers rather than online proctoring because it’s what they have been doing for a long time, rather than necessarily because it is safer.

Undercover picture fromm BBC of test cheatingThere is a long history of security breaches at test centers, and some of the most flagrant happen due to proctors or invigilators helping participants pass the test. The BBC recently reported on several cases of cheating within construction industry exams at test centers in the UK – see here for a text summary and here for an excellent 13 minute video including undercover footage of the cheating. In one case, a proctor took the exam for the participant; in another case, a roomful of people were taking the exam and someone was showing the answers on a large screen at the front of the room (see picture to the right); and in a third case, participants answered the exam but didn’t submit, and the proctor reviewed and improved the answers before submitting.

This is just one of many, many examples. When participants see a financial benefit to passing a high stakes test, there are challenges in maintaining the integrity of a face-to-face proctoring system. It’s not impossible for online proctors to be corrupt, but when separated by distance, it is much harder.

The testing industry is very conservative. There will still be exams taken in test centers for many years to come. But I do think that 2016 may well be the beginning of the end for “bricks and mortar” test centers — and that during 2016, the mainstream view will start to think that online proctoring is the future.

Interested in learning more about online proctoring? I will be presenting a session on Ensuring Exam Integrity with Online Proctoring at the Questionmark Conference 2016: Shaping the Future of Assessment in Miami, April 12-15. I look forward to seeing you there! Click here to register and learn more about this important learning event. 

For more on online proctoring, check out this informational page and video below:

Checklists for Test Development

Austin Fossey-42Posted by Austin Fossey

There are many fantastic books about test development, and there are many standards systems for test development, such as The Standards for Educational and Psychological Testing. There are also principled frameworks for test development and design, such as evidence-centered design (ECD). But it seems that the supply of qualified test developers cannot keep up with the increased demand for high-quality assessment data, leaving many organizations to piece together assessment programs, learning as they go.checklist

As one might expect, this scenario leads to new tools targeted at these rookie test developers—simplified guidance documents, trainings, and resources attempting to idiot-proof test development. As a case in point, Questionmark seeks to distill information from a variety of sources into helpful, easy-to-follow white papers and blog posts. At an even simpler level, there appears to be increased demand for checklists that new test developers can use to guide test development or evaluate assessments.

For example, my colleague, Bart Hendrickx, shared a Dutch article from the Research Center for Examination and Certification (RCEC) at University of Twente describing their Beoordelingssysteem. He explained that this system provides a rubric for evaluating education assessments in areas like representativeness, reliability, and standard setting. The Buros Center for Testing addresses similar needs for users of mental assessments. In the Assessment Literacy section of their website, Buros has documents with titles like “Questions to Ask When Evaluating a Test”—essentially an evaluation checklist (though Buros also provides their own professional ratings of published assessments). There are even assessment software packages that seek to operationalize a test development checklist by creating a rigid workflow that guides the test developer through different steps of the design process.

The benefit of these resources is that they can help guide new test developers through basic steps and considerations as they build their instruments. It is certainly a step up from a company compiling a bunch of multiple choice questions on the fly and setting a cut score of 70% without any backing theory or test purpose. On the other hand, test development is supposed to be an iterative process, and without the flexibility to explore the nuances and complexities of the instrument, the results and the inferences may fall short of their targets. An overly simple, standardized checklist for developing or evaluating assessments may not consider an organization’s specific measurement needs, and the program may be left with considerable blind spots in its validity evidence.

Overall, I am glad to see that more organizations are wanting to improve the quality of their measurements, and it is encouraging to see more training resources to help new test developers tackle the learning curve. Checklists may be a very helpful tool for a lot of applications, and test developers frequently create their own checklists to standardize practices within their organization, like item reviews.

What do our readers think? Are checklists the way to go? Do you use a checklist from another organization in your test development?





9 trends in compliance learning, training and assessment

John Kleeman HeadshotThis version is a re-post of a popular blog by John Kleeman

Where is the world of compliance training, learning and assessment going?

I’ve collaborated recently with two SAP experts, Thomas Jenewein of SAP and Simone Buchwald of EPI-USE, to write a white paper on “How to do it right – Learning, Training and Assessments in Regulatory Compliance[Free with registration]. In it, we suggested 9 key trends in the area. Here is a summary of the trends we see:

1. Increasing interest in predictive or forward-looking measures

Many compliance measures (for example, results of internal audits or training completion rates) are backwards looking. They tell you what happened in the past but don’t tell you about the problems to come. Companies can see clearly what is in their rear-view mirror, but the picture ahead of them is rainy and unclear. There are a lot of ways to use learning and assessment data to predict and look forward, and this is a key way to add business value.

2. Monitoring employee compliance with policies

A recent survey of chief compliance officers suggested that their biggest operational issue is monitoring employee compliance with policies, with over half of organizations raising this as a concern. An increasing focus for many companies is going to be how they can use training and assessments to check understanding of policies and to monitor compliance.

3. Increasing use of observational assessments

Picture of observational assessment on smartphoneWe expect growing use of observational assessments to help confirm that employees are following policies and procedures and to help assess practical skills. Readers of this blog will no doubt be familiar with the concept. If not, see Observational Assessments—why and how.

4. Compliance training conducted on mobile devices

The world is moving to mobile devices and this of course includes compliance training and assessment.

5. Informal learning

You would be surprised not to see informal learning in our list of trends. Increasingly we are all understanding that formal learning is the tip of the iceberg and that most learning is informal and often on the job.

6. Learning in the extended enterprise

Organizations are becoming more interlinked, and another important trend is the expansion of learning to the extended enterprise, such as contractors or partners. Whether for data security, product knowledge, anti-bribery or a host of other regulatory compliance reasons, it’s becoming crucial to be able to deliver learning and to assess not only your employees but those of other organizations who work closely with you.

7. Cloud

There is a steady movement towards the cloud and SaaS for compliance learning, training, and assessment – with the huge advantage of delegating all of the IT to an outside party being the strongest compelling factor.  Especially for compliance functions, the cloud offers a very flexible way to manage learning and assessment without requiring complex integrations or alignments with a company’s training departments or related functions.

8. Changing workforce needs

The workforce is constantly changing, and many “digital natives” are now joining organizations. To meet the needs of such workers, we’re increasingly seeing “gamification” in compliance training to help motivate and connect with employees. And the entire workforce is now accustomed to seeing high-quality user interfaces in consumer Web sites and expects the same in their corporate systems.

9. Big Data

E-learning and assessments are a unique way of touching all your employees. There is huge potential in using analytics based on learning and assessment data. We have the potential to combine Big Data available from valid and reliable learning assessments with data from finance, sales, and HR sources.  See for example the illustration below from SAP BusinessObjects showing assessment data graphed against performance data as an illustration of what can be done.

data exported using OData from Questionmark into SAP BusinessObjects

For information on these trends, see the white paper written with SAP and EPI-USE: “How to do it right – Learning, Training and Assessments in Regulatory Compliance”, available free to download with registration.

If you have other suggestions for trends, feel free to contribute them below.

Ten tips for securing your assessment system

Picture of key amongst numbers to illustrate cyber securityJohn Kleeman HeadshotPosted by John Kleeman

What can you do to make your assessment system more secure? How can you avoid a disruptive data breach where people’s personal information is disclosed? Using a vendor who takes security seriously reduces risk, as I wrote in  my blog article Eight ways to check if security is more than skin deep. But security involves both vendor and user. This post gives ten good practice tips on how you as a user or administrator of an assessment system can reduce the risk of data breaches.

1. Don’t give yourself or other administrators unnecessary privileges. Follow the principle of least privilege. It may sound counter-intuitive, but most administrative users don’t need access to all capabilities and data within your system. Limiting access reduces the impact of a data breach if an account is compromised or someone makes a mistake. If you are using Questionmark, allocate appropriate roles to limit people to what they need.

2. When someone leaves the project or organization, remove their access. Don’t allow someone who has left your team to still have access to your assessment data.

3. Follow good password security. Do not share passwords between people. Do not use the same password for two accounts. Choose strong passwords and change them periodically.  If someone asks you for your password, never, ever give it. And if a web page doesn’t look right, don’t type your password into it.

4. Install all the patches and secure the system. A common cause of security breaches is failing to install the latest versions of software, and attackers exploit known vulnerabilities. You need to be proactive and always install the latest version of system and application software, set up good technical security and follow the vendor’s recommendations.

If you haven’t got the time or resources to do this properly, move to a cloud solution. In a cloud SaaS solution like Questionmark OnDemand, the vendor is responsible for updating Windows, updating the application, monitoring security and ensuring that everything is up to date.

5. Install good quality antivirus / anti-malware software.  Reportedly there are nearly a million new or variant  malware and viruses produced each day. Protect your computer and those of your co-workers with up to date, professional software to address this threat.

6. Protect any downloaded data. Questions, assessments and reports on results are generally safer on a server or in an on-demand service than on a workstation. If you need to download data locally, set up security procedures to protect it and try to ensure that any download is temporary only.

7. Dispose of data properly. Deleting a file on a computer doesn’t erase the data, it simply erases the index to it. If you use a reputable service like Questionmark OnDemand, if a disk is repaired or reaches end of life, it will be securely destroyed for example by degaussing. But if you download data locally or use installable software to manage your assessments, you need to do this yourselves. A recent study suggested that about half of used hard drives sold online contain residual data. Make sure this is not your assessment data!

8. Be careful about clicking on a link or attachment in an email. Phishing attacks use email or malicious websites (clicking on a link) to collect sensitive information or infect your machine with malware and viruses. Such attacks could even be aimed at your organization or assessment activity directly (this is called spear phishing!). Think before clicking.

9. Be aware of social engineering.  Social engineering is when someone tries to trick you or someone else into a security breach. For example someone might ring up and claim to be a student who wants their results, but really is an imposter. Or someone might spoof an email from your boss asking for the questions for the next test to review. Be wary of strange phone calls or emails that ask for something urgent. If something seems suspicious, clear it with a security professional before you give them info or ask a caller to hang up and call them back on an official number.

10. Conduct security awareness training. If you’re not already doing this, organize training sessions for all your authors, proctors, administrators and other users to help them be security aware. if you can, deliver tests after the training to check understanding. Sharing this blog article with your co-workers would be a great way to start.

Dance Video, Deadlines and Agenda – oh my!

The 2016 Questionmark Conference will be a seriously fun and jam packed learning event in Miami April 12-15.miami-vertical-collage2

sunWhat’s on the agenda? Here’s a sneak-peek of what you can expect:

  • Secure Authentication: Accessing Questionmark OnDemand with SSO
  • Role-based Security in Questionmark OnDemand
  • Taking Your Test Planning to the Next Level: JTAs and Blueprints 
  • Capturing Demographics When Delivering Assessments

sunAre you a Questionmark superstar? Share your story in Miami!

  • Do you have an experience you would like to share, or a topic to discuss? Submit your proposal by Friday (December 4) and become a speaker at the Conference. *Slots are limited
  • Check out this dance video and get in the mood to learn and network alongside your peers in sunny Miami!

Invite Video