Six tips to increase content validity in competence tests and exams

Posted by John Kleeman

Content validity is one of the most important criteria on which to judge a test, exam or quiz. This blog post explains what content validity is, why it matters and how to increase it when using competence tests and exams within regulatory compliance and other work settings.

What is content validity?

An assessment has content validity if the content of the assessment matches what is being measured, i.e. it reflects the knowledge/skills required to do a job or demonstrate that the participant grasps course content sufficiently.

Content validity is often measured by having a group of subject matter experts (SMEs) verify that the test measures what it is supposed to measure.

Why does content validity matter?

If an assessment doesn’t have content validity, then the test isn’t actually testing what it seeks to, or it misses important aspects of job skills.

Would you want to fly in a plane, where the pilot knows how to take off but not land? Obviously not! Assessments for airline pilots take account all job functions including landing in emergency scenarios.

Similarly, if you are testing your employees to ensure competence for regulatory compliance purposes, or before you let them sell your products, you need to ensure the tests have content validity – that is to say they cover the job skills required.

Additionally to these common sense reasons, if you use an assessment without content validity to make decisions about people, you could face a lawsuit. See this blog post which describes a US lawsuit where a court ruled that because a policing test didn’t match the job skills, it couldn’t be used fairly for promotion purposes.

How can you increase content validity?

Here are some tips to get you started. For a deeper dive, Questionmark has several white papers that will help, and I also recommend Shrock & Coscarelli’s excellent book “Criterion-Referenced Test Development”.

  1. Conduct a job task analysis (JTA). A JTA is a survey which asks experts in the job role what tasks are important and how often they are done. A JTA gives you the information to define assessment topics in terms of what the job needs. Questionmark has a JTA question type which makes it easy to deliver and report on JTAs.
  2. Define the topics in the test before authoring. Use an item bank to store questions, and define the topics carefully before you start writing the questions. See Know what your questions are about before you deliver the test for some more reasoning on this.
  3. You can poll subject matter experts to check content validity for an existing test. If you have an existing assessment, and you need to check its content validity, get a panel of SMEs (experts) to rate each question as to whether it is  “essential,” “useful, but not essential,” or “not necessary” to the performance of what is being measured. The more SMEs who agree that items are essential, the higher the content validity. See Understanding Assessment Validity- Content Validity for a way to do this within Questionmark software.
  4. Use item analysis reporting. Item analysis reports flag questions which are don’t correlate well with the rest of the assessment. Questionmark has an easy to understand item analysis report which will flag potential questions for review. One of the reasons a question might get flagged is because participants who do well on other questions don’t do well on this question – this could indicate the question lacks content validity.
  5. Involve Subject Matter Experts (SMEs). It might sound obvious, but the more you involve SMEs in your assessment development, the more content validity you are likely to get. Use an assessment management system which is easy for busy SMEs to use, and involve SMEs in writing and reviewing questions.
  6. Review and update tests frequently. Skills required for jobs change quickly with changing technology and changing regulations.  Many workplace tests that were valid two years ago, are not valid today. Use an item bank with a search facility to manage your questions, and review and update or retire questions that are no longer relevant.

I hope this blog post reminds you why content validity matters and gives helpful tips to improve the content validity of your tests. If you are using a Learning Management System to create and deliver assessments, you may struggle to obtain and demonstrate content validity. If you want to see how Questionmark software can help manage your assessments, request a personalized demo today.

 

Questionmark is ISO 27001 certified. What this means for you?

Posted by John Kleeman

As you may have seen on our news site, Questionmark has just been certified to the Information Security standard ISO 27001.

What is ISO 27001?Chart showing probability and impact of risks, with high impact and high probability risks in red

ISO 27001 (full name ISO/IEC 27001:2013) is an international information security standard that is widely recognized as credible and authentic in validating that the certified organization has an effective management system for security.

The core of ISO 27001 is risk management. You identify in a systematic way risks to confidentiality, integrity, and availability and then assess their impact and probability. As simplistically shown in the diagram to the right, you decide what risks you can accept and how you can mitigate or otherwise deal with those that you cannot accept.

Subject to risk assessment, ISO 27001 requires you to meet over 100 controls including all the usually expected security controls. It also requires top management commitment and very specific processes to deal with issues that arise and auditing and much more.

ISO 27001 also encourages continual improvement – with all the threats out there, you have to keep making your processes and security better.

Questionmark’s ISO 27001 journey

Security has been central to Questionmark’s mission for decades. We brought out the world’s first secure browser in the 1990s, and our very first post in this blog back in 2009 was about delivering assessments safely and securely.

Last year, we decided that if we were to get external audit and validation of our security, it would both help us become more secure and help customers and other stakeholders feel more comfortable with our service. We’d been aware of 27001 for some time as the most credible security standard out there, and decided to adapt our processes and internal documentation to meet it. And we commissioned BSI, who are leaders in this field, to audit us.

The process to become certified is quite arduous. Including “internal” audits by a consultant and BSI’s audits, we have had eight days of auditing in the last few months. And these can be quite grueling – one of our audit days started with breakfast at 7 am and the auditor left the building just after 7.30pm at night! This definitely puts your people, processes, and technology through their paces. Implementing 27001 has improved Questionmark security and I’d encourage you to respect any organization who is certified as it’s a very credible process.BSI Assurance Mark Template RGB

I’m pleased to let you know that we are now certified by BSI under ISO 27001. Our certificate number is IS 668255. Our scope and certificate of applicability are wide, and we’d be pleased to share these with stakeholders under NDA.

How might it matter to purchasers of assessment services?

ISO 27001 certification gives external validation that an organization has a good quality information security management system.

Anyone can claim to be secure. Anyone can claim to follow standards. It’s hard for someone who is not a security expert to know whether an organization actually has put the effort into people, process, and technology to do the best that can be done to resist threats to confidentiality, integrity and availability.

With all the threats out there to assessment data, we believe it’s helpful to our customers to have assurance that Questionmark has been independently audited and it has been certified that our information security management system complies with ISO 27001.

How could ISO 27001 help assessment providers?

Are you looking to create and deliver secure assessments and keen to protect confidentiality, integrity, and availability?

Although using Questionmark OnDemand will help you do this, I’d encourage some blog readers to think whether it might make sense to implement ISO 27001 yourselves as an organization. That way you will ensure that all your IT and systems are securely managed. There are some work and effort involved, but it will make you as an organization more secure and less likely to suffer breaches and other failures.

I’ve just led Questionmark’s implementation of ISO 27001 and would be happy to share experiences with others in the assessment industry, please feel free to reach out to me.

Seven tips to recruit and manage SMEs for technology certification exams

imagePosted by John Kleeman

[repost from February 8, 2017]

How do you keep a certification exam up to date when the technology it is assessing is changing rapidly?

Certifications in new technologies like software-as-a-service and cloud solutions have some specific challenges. The nature of the technology usually means that questions often require very specialist knowledge to author. And because knowledge of the new technology is in short supply, subject matter experts (SMEs) who are able to author and review new items will be in high demand within the organization for other purposes.

Cloud technological offerings also change rapidly. It used to be that new technology releases came out every year or two, and if you were writing certification exams or other assessments to test knowledge and skill in them, you had plenty of notice and could plan an update cycle. But nowadays most technology organizations adopt an agile approach to development with the motto “release early, release often”. The use of cloud technology makes frequent, evolutionary releases – often monthly or quarterly – normal.

So how can you keep an exam valid and reliable if the content you are assessing is changing rapidly?

Here are seven tips that could help – a few inspired by an excellent presentation by Cisco and Microsoft at the European Association of Test Publishers conference.

  1. Try to obtain item writing SMEs from product development. They will know what is coming and what is changing and will be in a good position to write accurate questions. 
  2. Also network for SMEs outside the organization – at technology conferences, via partners and resellers, on social media and/or via an online form on your certification website. A good source of SMEs will be existing certified people.
  3. Incentivize SMEs – what will work best for you will depend on your organization, but you can consider free re-certifications, vouchers, discounts off conferences, books and other incentives. Remember also that for many people working in technology, recognition and appreciation are as important as financial incentives. Appreciate and recognize your SMEs. For internal SMEs, send thank you letters to their managers to appreciate their effort.
  4. Focus your exam on underlying key knowledge and skills that are not going to become obsolete quickly. Work with your experts to avoid items that are likely to become obsolete and seek to test on fundamental concepts, not version specific features.
  5. When working with item writers, don’t be frightened to develop questions based on beta or planned functionality, but always do a check before questions go live in case the planned functionality hasn’t been released yet.
  6. Analyze, create, deliverSince your item writers will likely be geographically spread and will be busy and tech-literate, use a good collaborative tool for item writing and item banking that allows easy online review and tracking of changes. (See https://www.questionmark.com/content/distributed-authoring-and-item-management for information on Questionmark’s authoring solution.)
  7. In technology as in other areas, confidentiality and exam security are crucial to ensure the integrity of the exam. You should have a formal agreement with internal and external SMEs who author or review questions to remind them not to pass the questions to others. Ensure that your HR or legal department are involved in the drafting of these so that they are enforceable.

Certification of new technologies helps adoption and deployment and contributes to all stakeholders success. I hope these tips help you improve your assessment program.

GDPR is coming. Are you ready?

Posted by Julie Delazyn

Don’t get left behind as the most important change in data privacy takes effect May 2018. The new General Data Protection Regulation (GDPR) intends to strengthen and unify privacy and data protection and any organization that stores or manages data about Europeans will need to comply.

With eye-watering regulatory fines of up to €20 million or 4% of global annual turnover (whichever is greater), a credible compliance strategy is essential.

Join us for a FREE 45 minute Webinar July 26, 2017, to understand how online assessments can help you meet your GDPR challenges.

The webinar will cover:

  • What the GDPR is and who it impacts
  • Why you should care about GDPR compliance
  • How to overcome the challenges presented by GDPR — including the learning curve for your employees
  • How assessment can help mitigate GDPR risks and aid your compliance strategy
  • Considerations for implementing assessment management software to aid in compliance

We look forward to speaking to you at the webinar!

The Power of Open: Questionmark’s open assessment platform

Posted by Steve Lay

In the beginning there was CVS, then there was SVN and now there’s Git.  What am I talking about?  These are all source code control systems, systems that are used to store computer source code in a way that preserves the complete version history and provides a full audit trail covering the who, what, when and why changes were made.

When we think of open source software we tend to think of the end product: a freely downloadable program that you can run on your computer or even a complete computer operating system in the case of Linux.  But to open source developers, open source is about more than this ‘free beer’ model of sharing software.  Open source software is shared at the source code level allowing people to examine the way it works, suggest changes to fix bugs, enhance it or even to modify it for their own purposes.  Getting the most from sharing source code requires more than just sharing an executable or a zip file of the finished product, open source developers need to open up their source code control systems too.

For years there have been services that provide a cloud-based alternative to  hosting your own source code.  The SourceForge system enjoyed many years of dominance but more recently it’s advertising sponsored model has seen it fall out of favour.

Most new projects are now created on a service called GitHub, which promises  free hosting of open source projects on a service funded by paying customers who are developing projects privately on the same platform.  The success of GitHub has been phenomenal – Google closed down its own rival service (Google Code) largely because of GitHub’s success.  In fact, GitHub is rapidly becoming a ‘unicorn’ with all the associated growing pains.  GitHub makes it easy to collaborate on projects too with its issue tracking system and user friendly tools for proposing changes (known as ‘pull requests’).

With GitHub as the de facto place to publish and share source code, it makes sense for Questionmark to use it to complement our Open Assessment Platform.  We have published source code illustrating how to use our APIs for many years and even publish the complete source to some of our connectors.  Putting new projects on GitHub means providing sample code in the most transparent and developer-friendly way possible.

Questionmark’s GitHub page lists all the projects we own.  For example, when we first brought out our OData APIs we published the sample reportlet code in the OData Reportlet Samples project.  You can experiment with these same examples running live in our website’s developer pages.

Recently we’ve gone a step further in opening up our assessment platform.  We’ve started publishing our API documentation via GitHub too!  Using a new feature of the GitHub platform we’re able to publish the documentation directly from the source control system itself.  That means you always get access to the latest documentation.

Opening up our API documentation in this way makes it easier for developers to engage with our platform.  Why not check out the documentation project.  If you’re already a GitHub user you could ‘watch’ it to get notified when we make changes.  You can even submit issues or send us ‘pull requests’ if you have suggestions for improvement.

With GitHub as the de facto place to publish and share source code, it makes sense for Questionmark to use it to complement our Open Assessment Platform.  We have published source code illustrating how to use our APIs for many years and even publish the complete source to some of our connectors.  Publishing this source code helps our customers and partners by providing working examples of how to integrate with our platform as well as providing complete transparency for our connectors allowing customers to audit the code before they run it on their own systems.  Putting new projects on GitHub means providing sample code in the most transparent and developer-friendly way possible.

How online assessments (quizzes, tests and exams) can help information security awareness and compliance

Posted by John Kleeman

With the rise of data security leakages, most professional organizations are seeking to significantly upscale their cybersecurity to better protect their organization from information security risks. I see an increasing use of online assessments helping information security and thought I’d provide some pointers about this.

There are three main ways in which online quizzes, tests, exams and surveys can aid information security:

  • Testing personnel to check understanding of security awareness and security policies
  • Ensuring and documenting that personnel in security roles are competent
  • Helping measure success against security objectivesNIST logo

Testing on security awareness and knowledge of policies

A cornerstone of good practice in security is training in security awareness. For example, the widely respected NIST 800-53 publication recommends that organizations provide general-purpose and role-based training to personnel as part of initial training and periodically thereafter. If you follow NIST standards, NIST control AT-4 also requires that all security training be documented and records retained.

There is widespread evidence that delivering an assessment is the best way of documenting that training took place, because it doesn’t just document attendance but also understanding of the training. For more explanation, see the Questionmark blog post Proving compliance – not just attendance. The only point of security awareness training is to have the training be understood, so testing to confirm understanding is widespread and sensible.

At Questionmark, we practice what we preach! All our employees have to take a test on data security when they join to check they understand our policies; all employees must also take and pass an updated test each year to ensure they continue to understand.

Ensure that people in security roles are competent

iso 27001The international security standard ISO 27001:2013 requires that an organization determine the necessary competence of personnel affecting information security performance. The organization must also ensures that personnel have such competence and retain evidence of this.

In a large organization with many different security roles, developing and using competence tests for each information security-related role is a good way of measuring and showing competence.  Knowing who is competent in which aspect of security and data protection matters: it ensures that  you are covering appropriate risks with appropriate people. Online testing is an effective way of measuring competence and makes it easy to update competence records by giving periodic tests every six months or annually.

Helping measure information security objectives

PCI logoISO 27001 also requires setting up metrics to measure information security objectives. Results from assessments can be a good metric to use.  Other standards say similar things. For example, the PCI standard widely used for credit card security says in its best practice guide:

“Metrics can be an effective tool to measure the success of a security awareness program, and can also provide valuable information to keep the security awareness program up-to-date and effective”

The PCI guide recognizes that good metrics include “feedback from personnel; quizzes and training assessments”. In my experience, as well as using quizzes and tests to measure knowledge, it also makes sense to use online surveys to assess actual practice by employees and to allow reporting of security concerns.

Testing on information security and data protection is an increasing use case for Questionmark’s trustable SaaS assessment management system, Questionmark OnDemand.  Whichever security standard you are following (ISO 27001, NIST, PCI or one of several others), creating online assessments tailored to measure knowledge of your organization’s policies and procedures using an assessment management system like Questionmark’s can make a useful difference.

Next Page »