7 ways assessments can save you money and protect your reputation [Compliance webinar]

Julie ProfilePosted by Julie Delazyn

Last week, illegal banking practices cost Wells Fargo, one of America’s largest banks, $185 million in fines. Regulators have called the scandal “outrageous” and stated that the widespread nature of the illegal behavior shows the bank lacked the necessary controls and oversight of its employees.

Educating and monitoring employee understanding of proper practices is vital for regulatory compliance.  How do you ensure your workers are compliant with the rules and regulations in your industry? How do you prove that employee training is understood?

Register today for the FREE webinar: 7 Ways Assessments Fortify Compliance

The webinar will examine real-world examples of how assessments are used to strengthen compliance programs. It will also provide tips for developing valid, reliable assessments.

The Importance of Safety in the Utilities Industry: A Q&A with PG&E

Headshot Julie

Posted by Julie Delazyn

Wendy Lau is a Psychometrician at Pacific Gas and Electric Company (PG&E). She will be leading a discussion at Questionmark Conference 2016 in Miami, about Safety and the Utilities Industry: Why Assessments Matter.

WendyLau_Q&A

Wendy Lau, Psychometrician, PG&E

Wendy’s session will describe a day in the life of a psychometrician in the utilities industry. It will explore the role assessments play at PG&E, and how Questionmark has helped the company focus on safety and train its employees.

I recently asked her about her session:

Tell me about PG&E and its use of assessments:

PG&E is a utilities company that provides natural gas and electricity to most of the northern two-thirds of California. Over the years, we have evolved into a more data-driven company, and Questionmark has been a part of that for the past 7 years. Having assessments readily available and secured within a platform that we can trust is very important to PG&E. We are also glad to have found a testing tool that offers such a wide variety of question types.

Why is safety important in the utilities industry?

Depending on the activity that our employees perform, most of the work has serious safety implications — whether it is a lineman climbing up a  pole to perform liveline work or a utility worker digging near a major gas pipeline. Our technical training must have safety in mind and, more importantly, it must ensure that after going through training, employees are competent to perform their tasks safely and proficiently. In order to ensure workforce capability, we rely heavily on testing to prove that our workforce is in fact safe and proficient and that the community we serve and our employees are safe and receiving reliable services.

What role does Questionmark play in ensuring that safety?

Questionmark helps us focus on safety-related questions by allowing special assessment strategies such as identifying critical versus coachable assessment items and identifying cutscores for each accordingly. Questionmark also allows a secured platform so that we can ensure our test items are never compromised and that our employees are truly being assessed under fair circumstances.

To find out more about the role of Questionmark plays in ensuring safety, you’ll just have to attend my session at Questionmark Conference 2016 in Miami!

What are you looking forward to at the conference?

I am very much looking forward to ‘talking shop’ with other Psychometricians and sharing best practices with others in the utilities industry and other companies alike!

Thank you, Wendy for taking time out of your busy schedules to discuss your session with us!

palm tree emoji 2If you have not already done so, you still have a chance to attend this important learning event. Click here to register.

 

Get tips for combatting test fraud

Chloe MendoncaPosted by Chloe Mendonca

There is a lot of research to support the fact that stepping up investment in learning, training and certification is critical to professional success. A projection from the Institute for Public Policy Research states that ‘between 2012 and 2022, over one-third of all jobs will be created in high-skilled occupations’. This growing need for high-skilled jobs is resulting in a rapid increase in professional qualifications and certifications.

Businesses are recognising the need to invest in skills, spending some £49 billion in 2011 alone on training [figures taken from CBI on skills] — and assessments are a big part of this. They have become widely adopted in helping to evaluate the competence, performance and potential of employees and job candidates. In many industries such as healthcare, life sciences and manufacturing, the stakes are high. Life, limb and livelihood are on the line, so delivering such assessments safely and securely is vital.

Sadly, many studies show that the higher the stakes of an assessment, the higher the potential and motivation to commit test fraud. We see many examples of content theft, impersonation and cheating in the news, so what steps can be taken to mitigate security risks?? What impact do emerging trends such as online remote proctoring have on certification programs? How can you use item banking, secure delivery apps and reporting tools to enhance the defensibility of your assessments?

This October, Questionmark will deliver breakfast briefings in two UK cities, providing the answers to these questions. The briefings will include presentations and discussions on the tools and practices that can be used to create and deliver secure high-stakes tests and exams.

These briefings, due to take place in London and Edinburgh, will be ideal for learning, training and compliance professionals who are using or thinking about using assessments. We invite you to find out more and register for one of these events:

 

Q&A: Pre-hire, new-hire and ongoing assessments at Canon

HollyGroder

Holly Groder

Headshot JuliePosted by Julie Delazyn

Holly Groder and Mark Antonucci are training developers for Canon Information Technology Services, Inc. (Canon ITS). During their case study presentation at the Questionmark 2015 Users Conference in Napa Valley March 10-13, they will talk about Leveraging Questionmark’s Reports and Analytics Tools for Deeper Insight.

Their session will explore Canon’s use of assessments in hiring, training, continuing job skills assessment and company-wide information gathering via surveys.

I asked them recently about their case study:

Why did you start using Questionmark? 

The primary reason for seeking a new assessment tool was our desire to collect more information from our assessments, quicker. Questionmark offered the flexibility of web-based question creation and built-in reports. Questionmark also offered the ability to add jump blocks and a variety of templates. The survey capabilities were just a bonus for us. We were able to streamline our survey process to one point of contact and eliminate an additional software program.

What kinds of assessments do you use?

MarkAntonucci1

Mark Antonucci

The principal function is split between four business needs: Pre-hire employment assessments, new-hire or cross-training assessments, continuing job knowledge assessments, and business information gathering (surveys).

How are you using those tools?

First, potential employees are required to participate in a technical knowledge assessment prior to an offer of employment. Once employment has been offered and accepted, the new employees are assessed throughout the new-hire training period. Annually, all call center agents participate in a job skills assessment unique to their department. And finally, all employees participate in various surveys ranging from interest in community events to feedback on peer performance.

What are you looking forward to at the conference?

We are interested in best practices, insight into psychometrics, and, most important, networking with other users.

Thank you Holly and Mark for taking time out of your busy schedules to discuss your session with us!

***

If you have not already done so, you still have a chance to attend this important learning event. Click here to register.

Item Development – Training Item Writers

Austin FosseyPosted by Austin Fossey

Once we have defined the purpose of the assessment, completed our domain analysis, and finalized a test blueprint, we might be eager to jump right in to item writing, but there is one important step to take before we begin: training!

Unless you are writing the entire assessment yourself, you will need a group of item writers to develop the content. These item writers are likely experts in their fields, but they may have very little understanding of how to create assessment content. Even if these experts have experience writing items, it may be beneficial to provide refresher trainings, especially if anything has changed in your assessment design.

In their chapter in Educational Measurement (4 th ed.), Cynthia Shmeiser and Catherine Welch note that it is important to consider the qualifications and representativeness of your item writers. It is common to ask item writers to fill out a brief survey to collect demographic information. You should keep these responses on file and possibly add a brief document explaining why you consider these item writers to be a qualified and representative sample.

Shmeiser and Welch also underscore the need for security. Item writers should be trained on your content security guidelines, and your organization may even ask them to sign an agreement stating that they will abide by those guidelines. Make sure everyone understands the security guidelines, and have a plan in place in case there are any violations.

Next, begin training your item writers on how to author items, which should include basic concepts about cognitive levels, drafting stems, picking distractors, and using specific item types appropriately. Shmeiser and Welch suggest that the test blueprint be used as the foundation of the training. Item writers should understand the content included in the specifications and the types of items they are expected to create for that content. Be sure to share examples of good and bad items.

If possible, ask your writers to create some practice items, then review their work and provide feedback. If they are using the item authoring software for the first time, be sure to acquaint them with the tools before they are given their item writing assignments.

Your item writers may also need training on your item data, delivery method, or scoring rules. For example, you may ask item writers to cite a reference for each item, or you might ask them to weight certain items differently. Your instructions need to be clear and precise, and you should spot check your item writers’ work. If possible, write a style guide that includes clear guidelines about item construction, such as fonts to use, acceptable abbreviations, scoring rules, acceptable item types, et cetera.

I know from my own experience (and Shmeiser and Welch agree) that investing more time in training will have a big payoff down the line. Better training leads to substantially better item retention rates when items are reviewed. If your item writers are not trained well, you may end up throwing out many of their items, which may not leave you enough for your assessment design. Considering the cost of item development and the time spent writing and reviewing items, putting in a few more hours of training can equal big savings for your program in the long run.

In my next post, I will discuss how to manage your item writers as they begin the important work of drafting the items.

Get trustable results: How many test or exam retakes should you allow?

John Kleeman HeadshotPosted by John Kleeman

How many times is it fair and proper for a participant to retake an assessment if they fail?

One of our customers asked me about this recently in regard to a certification exam. I did some research and thought I’d share it  here.

For a few kinds of assessments, you would normally only allow a single attempt, typically if you are measuring something at a specific point in time. A pre-course or post-course test might only be useful if it is taken right before or right after a training course.

Regarding assessments that just give retrieval practice or reinforce learning, you needn’t be concerned. It may be fine to allow as many retakes as people want. The more times they practice answering the questions, the more they will retain the learning.

But how can you decide how many attempts to allow at a certification assessment measuring competence and mastery?

Consider test security

Retakes can jeopardize test security. Someone might take and retake a test to harvest the items to share with others. The more retakes allowed, the more this risk increases.

International Test Commission draft security guidelines say:

“Retake policies should be developed to reduce the opportunities for item harvesting and other forms of test fraud. For example, a test taker should not be allowed to retake a test that he or she “passed” or retake a test until a set amount of time has passed.”

Consider measurement error

All assessment scores have measurement error. A certification exam classifies people as having mastery (pass) or not (failing), but it doesn’t do so perfectly.

If you allow repeat retakes, you increase the risk of classifying someone as a master who is not competent, but  you also decrease the risk of classifying a competent person as having failed. This is because someone can suffer test anxiety or be ill or make a stupid mistake and fail the test despite being competent.

Require participants to wait for retakes

It’s usual to require a time period to elapse before a retake. This  stops people from  using quick, repeated retakes to take unfair advantage of measurement error. It also encourage reflection and re-learning before the next attempt. Standard 13.6 in the Standards for Educational and Psychological Testing says:

“students. . . should have a reasonable number of opportunities to succeed. . . the time intervals between the opportunities should allow for students to have the opportunity to obtain the relevant instructional experiences.”

If we had a perfectly reliable assessment, there would be no concern about multiple attempts. Picking the number of attempts is a compromise between what is fair to the participants and the limitations of our resources as assessment developers.

Think about test preparation

Could your retake policy affect how people prepare for the exam?

If retakes are easily available, some participants might prepare less effectively, hoping that they can “wing it” since  they can retake at will.  On the other hand, if retakes are limited, this could increase test anxiety and stress. It could also increase the motivation to cheat.

What about fairness?

Some people suffer test anxiety, some people make silly mistakes on the test or use poor time management, and some may be not at their full capacity on the day of the exam. It’s usually fair to offer a retake in such situations. If you do not offer sufficient opportunities to retake, this will impact the face validity of the assessment: people might not consider it fair.

If your exam is open the public, you may not be able to limit retakes. Imagine a country where you were not allowed to retake your driving test once you’d failed it 3 times! It might make the roads safer, but most people wouldn’t see it as equitable.

In my next post on this subject, I will share what some organizations do in practice and offer some steps for arriving at an answer that will be suitable for your organization.