FAQ – “Testing Out” of Training

Posted by Kristin Bernor

Let’s explore what it means to “test out”, what the business benefits include and how Questionmark enables you to do this in a simple, timely and valid manner.

“Testing out” of training saves time and money by allowing participants to forego unneeded training. It makes training more valid and respected, and so more likely to impact behavior, because it focuses training on the people who need it and further allows those that do know it, to learn additional knowledge, skills and abilities.

The key to “testing out” of training is that the test properly measures what it is you are training. If that is the case, then if someone can demonstrate by passing the test that they know it already, then they don’t need to do the training. Why testing out can sometimes be a hard sell is if the test doesn’t really measure the same outcomes as the training – so just because you pass the test, you might not in fact know the training. So, the key is to write a good test.

Online assessments are about both staying compliant with regulatory requirements AND giving business value. Assessments help ensure your workforce is competent and reduce risk, but they also give business value in improved efficiency, knowledge and customer service.

What does it mean to “test out” of training?

Many organizations create tests that allow participants to “test out” of training if they pass. Essentially, if you already know the material being taught, then you don’t need to spend time in the training. Testing them on training that is already know is a waste of time, value and resources. Directing them to training that is necessary ensures the candidate is motivated and feels they are spending their time wisely. Everyone wins!

Why is this so important? Or What are the advantages to incorporating “testing out”?

The key advantage of this approach is that you save time when people don’t have to attend the training that they don’t need. Time is money for most organizations, and saving time is an important benefit.

Suppose, for example, you have 1,000 people who need to take some training that lasts 2 hours. This is 2,000 hours of people’s time. Now, suppose you can give a 20-minute test that 25% of people pass and therefore skip the training. The total time taken is 333 hours for the test and 1,500 hours for the training, which adds up to 1,833 hours. So having one-fourth of the test takers skip the training saves 9% of the time that would have been required for everyone to attend the training.

In addition to saving time, using diagnostic tests in this way helps people who attend training courses focus their attention on areas they don’t know well and be more receptive to the training that is beneficial.

Is it appropriate to allow “testing out” of all training?

Obviously if you follow this approach, you’ll need to ensure that your tests are appropriate and sufficient – that they measure the right knowledge and skills that the training would otherwise cover.

You’ll need to check your regulations to confirm that this is permissible for you, but most regulators will see sense here.

How Questionmark can be used to “test out”

Online assessments are a consistent and cost-effective means of validating that your workforce knows the law, your procedures and your products. If you are required to document training, it’s the most reliable way of doing so. When creating and delivering assessments within Questionmark, it’s quite simple to qualify a candidate once they reach a score threshold. If they correctly answer a series of items and pass the assessment, this denotes that further training is not needed. It is imperative that the assessment accurately tests for the requisite knowledge that are part of the training objectives.

The candidate can then focus on training that is pertinent, worthwhile and beneficial to both themselves and the company. If they answer incorrectly and are unable to pass the assessment, then training is necessary until they are able to master the information and demonstrate this in a test.

7 ways assessments can save you money and protect your reputation [Compliance webinar]

Julie ProfilePosted by Julie Delazyn

Last week, illegal banking practices cost Wells Fargo, one of America’s largest banks, $185 million in fines. Regulators have called the scandal “outrageous” and stated that the widespread nature of the illegal behavior shows the bank lacked the necessary controls and oversight of its employees.

Educating and monitoring employee understanding of proper practices is vital for regulatory compliance.  How do you ensure your workers are compliant with the rules and regulations in your industry? How do you prove that employee training is understood?

Register today for the FREE webinar: 7 Ways Assessments Fortify Compliance

The webinar will examine real-world examples of how assessments are used to strengthen compliance programs. It will also provide tips for developing valid, reliable assessments.

The Importance of Safety in the Utilities Industry: A Q&A with PG&E

Headshot Julie

Posted by Julie Delazyn

Wendy Lau is a Psychometrician at Pacific Gas and Electric Company (PG&E). She will be leading a discussion at Questionmark Conference 2016 in Miami, about Safety and the Utilities Industry: Why Assessments Matter.

WendyLau_Q&A

Wendy Lau, Psychometrician, PG&E

Wendy’s session will describe a day in the life of a psychometrician in the utilities industry. It will explore the role assessments play at PG&E, and how Questionmark has helped the company focus on safety and train its employees.

I recently asked her about her session:

Tell me about PG&E and its use of assessments:

PG&E is a utilities company that provides natural gas and electricity to most of the northern two-thirds of California. Over the years, we have evolved into a more data-driven company, and Questionmark has been a part of that for the past 7 years. Having assessments readily available and secured within a platform that we can trust is very important to PG&E. We are also glad to have found a testing tool that offers such a wide variety of question types.

Why is safety important in the utilities industry?

Depending on the activity that our employees perform, most of the work has serious safety implications — whether it is a lineman climbing up a  pole to perform liveline work or a utility worker digging near a major gas pipeline. Our technical training must have safety in mind and, more importantly, it must ensure that after going through training, employees are competent to perform their tasks safely and proficiently. In order to ensure workforce capability, we rely heavily on testing to prove that our workforce is in fact safe and proficient and that the community we serve and our employees are safe and receiving reliable services.

What role does Questionmark play in ensuring that safety?

Questionmark helps us focus on safety-related questions by allowing special assessment strategies such as identifying critical versus coachable assessment items and identifying cutscores for each accordingly. Questionmark also allows a secured platform so that we can ensure our test items are never compromised and that our employees are truly being assessed under fair circumstances.

To find out more about the role of Questionmark plays in ensuring safety, you’ll just have to attend my session at Questionmark Conference 2016 in Miami!

What are you looking forward to at the conference?

I am very much looking forward to ‘talking shop’ with other Psychometricians and sharing best practices with others in the utilities industry and other companies alike!

Thank you, Wendy for taking time out of your busy schedules to discuss your session with us!

palm tree emoji 2If you have not already done so, you still have a chance to attend this important learning event. Click here to register.

 

Get tips for combatting test fraud

Chloe MendoncaPosted by Chloe Mendonca

There is a lot of research to support the fact that stepping up investment in learning, training and certification is critical to professional success. A projection from the Institute for Public Policy Research states that ‘between 2012 and 2022, over one-third of all jobs will be created in high-skilled occupations’. This growing need for high-skilled jobs is resulting in a rapid increase in professional qualifications and certifications.

Businesses are recognising the need to invest in skills, spending some £49 billion in 2011 alone on training [figures taken from CBI on skills] — and assessments are a big part of this. They have become widely adopted in helping to evaluate the competence, performance and potential of employees and job candidates. In many industries such as healthcare, life sciences and manufacturing, the stakes are high. Life, limb and livelihood are on the line, so delivering such assessments safely and securely is vital.

Sadly, many studies show that the higher the stakes of an assessment, the higher the potential and motivation to commit test fraud. We see many examples of content theft, impersonation and cheating in the news, so what steps can be taken to mitigate security risks?? What impact do emerging trends such as online remote proctoring have on certification programs? How can you use item banking, secure delivery apps and reporting tools to enhance the defensibility of your assessments?

This October, Questionmark will deliver breakfast briefings in two UK cities, providing the answers to these questions. The briefings will include presentations and discussions on the tools and practices that can be used to create and deliver secure high-stakes tests and exams.

These briefings, due to take place in London and Edinburgh, will be ideal for learning, training and compliance professionals who are using or thinking about using assessments. We invite you to find out more and register for one of these events:

 

Q&A: Pre-hire, new-hire and ongoing assessments at Canon

HollyGroder

Holly Groder

Headshot JuliePosted by Julie Delazyn

Holly Groder and Mark Antonucci are training developers for Canon Information Technology Services, Inc. (Canon ITS). During their case study presentation at the Questionmark 2015 Users Conference in Napa Valley March 10-13, they will talk about Leveraging Questionmark’s Reports and Analytics Tools for Deeper Insight.

Their session will explore Canon’s use of assessments in hiring, training, continuing job skills assessment and company-wide information gathering via surveys.

I asked them recently about their case study:

Why did you start using Questionmark? 

The primary reason for seeking a new assessment tool was our desire to collect more information from our assessments, quicker. Questionmark offered the flexibility of web-based question creation and built-in reports. Questionmark also offered the ability to add jump blocks and a variety of templates. The survey capabilities were just a bonus for us. We were able to streamline our survey process to one point of contact and eliminate an additional software program.

What kinds of assessments do you use?

MarkAntonucci1

Mark Antonucci

The principal function is split between four business needs: Pre-hire employment assessments, new-hire or cross-training assessments, continuing job knowledge assessments, and business information gathering (surveys).

How are you using those tools?

First, potential employees are required to participate in a technical knowledge assessment prior to an offer of employment. Once employment has been offered and accepted, the new employees are assessed throughout the new-hire training period. Annually, all call center agents participate in a job skills assessment unique to their department. And finally, all employees participate in various surveys ranging from interest in community events to feedback on peer performance.

What are you looking forward to at the conference?

We are interested in best practices, insight into psychometrics, and, most important, networking with other users.

Thank you Holly and Mark for taking time out of your busy schedules to discuss your session with us!

***

If you have not already done so, you still have a chance to attend this important learning event. Click here to register.

Item Development – Training Item Writers

Austin FosseyPosted by Austin Fossey

Once we have defined the purpose of the assessment, completed our domain analysis, and finalized a test blueprint, we might be eager to jump right in to item writing, but there is one important step to take before we begin: training!

Unless you are writing the entire assessment yourself, you will need a group of item writers to develop the content. These item writers are likely experts in their fields, but they may have very little understanding of how to create assessment content. Even if these experts have experience writing items, it may be beneficial to provide refresher trainings, especially if anything has changed in your assessment design.

In their chapter in Educational Measurement (4 th ed.), Cynthia Shmeiser and Catherine Welch note that it is important to consider the qualifications and representativeness of your item writers. It is common to ask item writers to fill out a brief survey to collect demographic information. You should keep these responses on file and possibly add a brief document explaining why you consider these item writers to be a qualified and representative sample.

Shmeiser and Welch also underscore the need for security. Item writers should be trained on your content security guidelines, and your organization may even ask them to sign an agreement stating that they will abide by those guidelines. Make sure everyone understands the security guidelines, and have a plan in place in case there are any violations.

Next, begin training your item writers on how to author items, which should include basic concepts about cognitive levels, drafting stems, picking distractors, and using specific item types appropriately. Shmeiser and Welch suggest that the test blueprint be used as the foundation of the training. Item writers should understand the content included in the specifications and the types of items they are expected to create for that content. Be sure to share examples of good and bad items.

If possible, ask your writers to create some practice items, then review their work and provide feedback. If they are using the item authoring software for the first time, be sure to acquaint them with the tools before they are given their item writing assignments.

Your item writers may also need training on your item data, delivery method, or scoring rules. For example, you may ask item writers to cite a reference for each item, or you might ask them to weight certain items differently. Your instructions need to be clear and precise, and you should spot check your item writers’ work. If possible, write a style guide that includes clear guidelines about item construction, such as fonts to use, acceptable abbreviations, scoring rules, acceptable item types, et cetera.

I know from my own experience (and Shmeiser and Welch agree) that investing more time in training will have a big payoff down the line. Better training leads to substantially better item retention rates when items are reviewed. If your item writers are not trained well, you may end up throwing out many of their items, which may not leave you enough for your assessment design. Considering the cost of item development and the time spent writing and reviewing items, putting in a few more hours of training can equal big savings for your program in the long run.

In my next post, I will discuss how to manage your item writers as they begin the important work of drafting the items.