Item Analysis for Beginners – Getting Started

Posted by John Kleeman
Do you use assessments to make decisions about people? If so, then you should regularly run Item Analysis on your results.  Item Analysis can help find questions which are ambiguous, mis-keyed or which have choices that are rarely chosen. Improving or removing such questions will improve the validity and reliability of your assessment, and so help you use assessment results to make better decisions. If you don’t use Item Analysis, you risk using poor questions that make your assessments less accurate.

Sometimes people can be fearful of Item Analysis because they are worried it involves too much statistics. This blog post introduces Item Analysis for people who are unfamiliar with it, and I promise no maths or stats! I’m also giving a free webinar on Item Analysis with the same promise.

An assessment contains many items (another name for questions) as figuratively shown below. You can use Item Analysis to look at how each item performs within the assessment and flag potentially weak items for review. By keeping only stronger questions in the assessment, the assessment will be more effective.

Picture of a series of items with one marked as being weak

Item Analysis looks at the performance of all your participants on the items, and calculates how easy or hard people find the items (“item difficulty” or “p-value”) and how well the scores on items correlate with or show a relationship with the scores on the assessment as a whole (“item discrimination” or correlation). Some of problematic questions that Item Analysis can identify are:

  • Questions almost all participants get right, and so which are very easy. You might want to review to these to see if they are appropriate for the assessment. See my earlier post Item Analysis for Beginners – When are very Easy or very Difficult Questions Useful? for more information.
  • Questions which are difficult, where a lot of participants get the questionwrong. You should check such questions in case they are mis-keyed or ambiguous.
  • Multiple choice questions where some choices are rarely picked. You might want to improve such questions to make the wrong choices more plausible.
  • Questions where there is a poor correlation between participants who get the question right and who do well on the assessment. For example it will flag questions that high performing participants perform poorly on. You should look at such questions in case they are ambiguous, mis-keyed or off-topic.

There is a huge wealth of information available in an Item Analysis report, and assessment experts will delve into the report in detail. But much of the key information in an Item Analysis report is useful to anyone creating and delivering quizzes, tests and exams.

The Questionmark Item Analysis report includes a graph which shows the difficulty of items compared against their discrimination, like in the example below. It flags questions by marking them amber or red if they fall into categories which may need review. For example, in the illustration below, four questions are marked in amber as having low discrimination and so potentially be worth looking at.

Illustration of Questionmark item analysis report showing some questions green and some amber

If you are running an assessment program, and not using Item Analysis regularly, then this throws doubt on the trustworthiness of your results. By using it to identify and improve weak questions you should be able to improve your validity and reliability.

Item Analysis is surprisingly effective in practice. I’m one of the team responsible at Questionmark for managing our data security test which all employees have to take annually to check their understanding of information security and data protection. We recently reviewed the test and ran Item Analysis and very quickly found a question with poor stats where the technology had changed but we’d not updated the wording, and another question where two of the choices could be considered right, which made it hard to answer. It made our review faster and more effective and helped us improve the quality of the test.

If you want to learn a little more about Item Analysis, I’m running a free webinar on the subject “Item Analysis for Beginners” on May 2nd. You can see details and register for the webinar at https://www.questionmark.com/questionmark_webinars. I look forward to seeing some of you there!

 

Seven Ways Assessments Fortify Compliance

Posted by John Kleeman
Picture of a tablet being used to take an assessment with currency symbols adjacentWhy do most of the world’s banks, pharmaceutical companies, utilities and other large companies use online assessments to test the competence of their employees?

It’s primarily because compliance fines round the world are high and assessments reduce the risk of regulatory compliance failures. Assessments also give protection to the organization in the event of an individual mis-step by proving that the organization had checked the individual’s knowledge of the rules prior to the mistake.

Here are seven reasons companies use assessments from my experience:

1. Regulators encourage assessments 

Some regulators require companies to test their workforce regularly. For example the US FDIC says in its compliance manual:

“Once personnel have been trained on a particular subject, a compliance officer should periodically assess employees on their knowledge and comprehension of the subject matter”

And the European Securities and Market Authority says in its guidelines for assessment of knowledge and competence:

“ongoing assessment will contain updated material and will test staff on their knowledge of, for example, regulatory changes, new products and services available on the market”

Other regulators focus more on companies ensuring that their workforce is competent, rather than specifying how companies ensure it, but most welcome clear evidence that personnel have been trained and have shown understanding of the training.

People sitting at desks with computers taking tests2. Assessments demonstrate commitment to your workforce and to regulators

Many compliance errors happen because managers pay lip service to following the rules but indicate in their behavior they don’t mean it. If you assess all employees and managers regularly, and require additional training or sanctions for failing tests, it sends a clear message to your workforce that knowledge and observance of the rules is genuinely required.

Some regulators also take commitment to compliance by the organization into account when setting the level of fines, and may reduce fines if there is serious evidence of compliance activities, which assessments can be a useful part of. For example the German Federal Court recently ruled that fines should be less if there is evidence of effective compliance management.

3. Assessments find problems early

Online assessments are one of the few ways in which a compliance team can touch all employees in an organization. You can see results by team, department, location or individual and identify who understands what and focus in on weak areas to look at improving. There is no better way to reach all employees.

4. Assessments document understanding after training

Many regulators require training to be documented. Giving someone an assessment after training doesn’t just confirm he or she attended the course but confirms they understood the training.

5. Assessments increase retention of knowledge and reduce forgetting

Can you remember everything you learned? Of course, none of us can!

There is good evidence that quizzes and tests increase retention and reduce forgetting. This is partly because people study for tests and so remind themselves of the knowledge they learned, which helps retain it. And it is partly because retrieving information in a quiz or test makes it easier to retrieve the same information in future, and so more likely to be able to apply in practice when needed.

6. By allowing testing out, assessments reduce the time and cost of compliance trainingTake test. If pass, skip training. Otherwise do training.

Many organizations permit employees to “test out” of compliance training. People can take a test and if they demonstrate good enough knowledge, they don’t need to attend the training. This concentrates training resources and employee time on areas that are needed, and avoids demoralizing employees with boring compliance training repeating what they already know.

7. Assessments reduce human error which reduces the likelihood of a compliance mis-step

Many compliance failures arise from human error. Root cause analysis of human error suggests that a good proportion of errors are caused by people not understanding training, training being missing or people not following procedures. Assessments can pick up and prevent mistakes caused by people not understanding what they should do or how to follow procedures, and so reduce the risk of error.

 

If you are interested in learning more about the reasons online assessments mitigate compliance risk, Questionmark are giving a webinar “Seven Ways Assessments Fortify Compliance” on April 11th. To register for this or our other free webinars, go to www.questionmark.com/questionmark_webinars.

2018 Questionmark Conference Recap – 2019 Announced

Posted by Kristin Bernor

Wow, just wow! The 2018 Questionmark User’s Conference inspired all that are new to Questionmark and those that have been long time customers. Being able to share experiences and use case scenarios was exhilarating. The three day learning experience was a deep dive into learning, fun, and networking – the enthusiasm was truly contagious. From the opening general session where new features and functions were released to the trolley ride over to the railroad museum where guests were greeted with amazing local food and music, this year’s conference was without a doubt the preeminent complete user’s group experience!

Amazing customer presenters were on hand to deliver testimonials to their peers on best practices and lessons learned to truly maximize the power of their assessments. We greatly appreciated case studies presented by our valuable customers from Progressive Insurance, American Institute of Certified Public Accountants, Southwest Gas, Intuitive Surgical, Rio Salado College and Caterpillar.

A lively panel discussion focused on transformation and growth and how assessments are integral to all organizations in order to succeed in those areas. Panel participants included Andrew Dominguez of Southwest Gas, Tricia Allen of Polycom, Bernt Nilsen of Norsk Test and Dave “Lefty” Lefkowith of Louisiana Department of Education. Their invaluable insights into creating high performing teams through assessments was an “aha moment” for many and we are truly grateful for their participation.

Engaging sessions that delved into best practices of using assessment to identify knowledge gaps, improve performance and make informed and defensible decisions were widespread. Questionmark Founder and Executive Director, John Kleeman, presented a session based on a recent white paper, “Assessing for Situational Judgment”. Another well attended session detailed extending your assessment management capabilities with Questionmark apps and left attendees excited to get back to the office and implement.

Evenings provided many opportunities for networking, fine dining and fun. Small group “dine-arounds” — a long-time, popular tradition at our annual conference — gave us all a great chance to take in the sights as we strolled through beautiful, historic Savannah. Delegates and staff attending our Thursday evening reception, hosted at the Georgia State Railroad Museum, enjoyed delicious food, great music and networking.

The conference closed with the unveiling of next year’s Questionmark Conference dates and location. Drum roll please…

Save the date for Questionmark Conference 2019 us in San Diego at the Hard Rock Hotel, February 26 – March 1, 2019 for even more learning, networking and fun!

Watch the conference recap below!!

Questionmark Recognized with Brandon Hall Award for Assessment Technology

Posted by Kristin Bernor

What an amazing way to start the year and what an honor! We were recognized by Brandon Hall Group with a “Best Advance in Assessment and Survey Technology” award as part of their Excellence in Technology Program. Questionmark was presented with this award at the Brandon Hall Group HCM Excellence Conference Awards Gala in Palm Beach Gardens, Florida on February 1, 2018.https://blog.questionmark.com/wp-content/uploads/2018/03/bhgaward-jan2018.jpg

“Questionmark goes beyond just capturing results and provides tools for analyzing, interpreting, and understanding the information that assessments are gathering. For organizations with regulatory, compliance, or certification needs, Questionmark provides unrivaled tools for testing and measuring results,” from David Wentworth, Principal Analyst of the Brandon Hall Group.

The award recognizes those organizations that have successfully deployed programs, strategies, modalities, processes, systems, and tools that have achieved measurable results. Criteria used to judge applicants included value proposition, product innovation, unique differentiators, product demonstration and measurable results.

See the press release here.

GDPR: 6 months to go

Posted by Jamie Armstrong

Anyone working with personal data, particularly in the European Union, will know that we are now just six months from “GDPR day” (as I have taken to calling it). On 25-May-2018, the EU General Data Protection Regulation (“GDPR”) will become applicable, ushering in a new privacy/data protection era with greater emphasis than ever on the rights of individuals when their personal data is used or stored by businesses and other organizations. In this blog post, I provide some general reminders about what the GDPR is and give some insight into Questionmark’s compliance preparations.

The GDPR replaces the current EU Data Protection Directive, which has been around for more than 20 years. To keep pace with technology advances and achieve greater uniformity on data protection, the EU began work on the GDPR over 5 years ago and finalized the text in April 2016. There then followed a period for regulators and other industry bodies to provide guidance on what the GDPR actually requires, to help organizations in their compliance efforts. Like all businesses that process EU personal data, whether based within the U.S., the EU or elsewhere, Questionmark has been busy in the months since the GDPR was finalized to ensure that our practices and policies align with GDPR expectations.

For example, we have recently made available revised versions of our EU OnDemand service and US OnDemand service terms and conditions with new GDPR clauses, so that our customers can be assured that their agreements with us meet data controller-data processor contract requirements. We have updated our privacy policy to make clearer what personal data we gather and how this is used when people visit and interact with our website. There is also a helpful Knowledge Base article on our website that describes the personal data Questionmark stores.

GDPR

One of the most talked-about provisions of the GDPR is Article 35, which deals with data protection impact assessments, or “DPIAs.” Basically, there is a requirement that organizations acting as data controllers of personal data (meaning that they determine the purpose and means of the processing of that data) complete a prior assessment of the impacts of processing that data if the processing is likely to result in a high risk to the rights and freedoms of data subjects. Organizations will need to make a judgment call regarding whether a high risk exists to require that a DPIA be completed. There are scenarios in which a DPIA will definitely be required, such as when data controllers process special categories of personal data like racial origin and health information, and in other cases some organizations may decide it’s safer to complete a DPIA even if not absolutely necessary to comply with the GDPR.

The GDPR expects that data processors will help data controllers with DPIAs. Questionmark has therefore prepared an example draft DPIA template that may be used for completing an assessment of data processing within Questionmark OnDemand. The draft DPIA template is available for download now.

In the months before GDPR day we will see more guidance from the Article 29 Working Party and national data protection authorities to assist organizations with compliance. Questionmark is committed to helping our customers being compliant with the GDPR and we’ll post more next year on this subject. We hope this update is useful in the meantime

Important disclaimer: This blog is provided for general information and interest purposes only, is non-exhaustive and does not constitute legal advice. As such, the contents of this blog should not be relied on for any particular purpose and you should seek the advice of their own legal counsel in considering GDPR requirements.

Badging and Assessment: If they know it, let them show it!

Posted by Brian McNamara

We are delighted to announce the availability of Questionmark Badging!

With Questionmark Badging and Questionmark OnDemand, you can grant “badges” to participants based on the outcomes achieved on assessments such as certification exams, post-course tests or advancement exams.Badges associated with Questionmark assessments provide participants with portable, verifiable digital credentials.

Badges aligned with Questionmark assessments can be tied in with competencies and achievements, helping organizations provide recognition and motivation for increasing knowledge and skills. For credentialing and awarding bodies, they can increase the visibility and value of certification programs.

The new app couples Questionmark’s capabilities in delivering valid, reliable and trustworthy assessments with the industry-leading digital credentialing platform from Credly. More than just a visual representation of accomplishment, digital badges provide participants with verifiable, portable credentials that can be shared and displayed across the web, including social networking sites such as LinkedIn, Facebook and Twitter.

Find more info about Questionmark Badging right here!