How do you improve exam integrity and candidate satisfaction?

Posted by Julie Delazyn

Proxy testing … Cheating …. Content theft. How can you be sure that the integrity of your exam isn’t compromised when the technologies and techniques are complex and continually evolving? Many organizations are increasingly turning to online proctoring solutions.

Research shows that when properly applied, online proctoring can be as or more effective than traditional test-center proctoring. Besides reducing security risks, it also improves candidate satisfaction by offering the convenience of taking tests from their homes, workplaces or colleges. It also reduces costs not only for administrators but also for test takers. Consider for a moment the testing fees, travel, parking, and the simple time away from work or studies.  In addition to online proctoring’s effectiveness, one of the most exciting things is that this approach has the potential to grow as technology becomes more advanced and widespread.

What do you need to consider before implementing online proctoring? What features are important to you? How will your exams retain their integrity? How are other colleges or credentialing organizations using online proctoring to improve candidate experience?

Join Questionmark for a 30-minute webinar answering these questions and more

The webinar will cover:

  • Record and review based system with automated flagging
  • What colleges, universities and credentialing organizations are using to offer 24/7 flexibility to test-takers
  • How to administer online exams in a fair way by combining test delivery with exam proctoring

Want to learn more? Secure your spot for this complimentary webinar

Six tips to increase content validity in competence tests and exams

Posted by John Kleeman

Content validity is one of the most important criteria on which to judge a test, exam or quiz. This blog post explains what content validity is, why it matters and how to increase it when using competence tests and exams within regulatory compliance and other work settings.

What is content validity?

An assessment has content validity if the content of the assessment matches what is being measured, i.e. it reflects the knowledge/skills required to do a job or demonstrate that the participant grasps course content sufficiently.

Content validity is often measured by having a group of subject matter experts (SMEs) verify that the test measures what it is supposed to measure.

Why does content validity matter?

If an assessment doesn’t have content validity, then the test isn’t actually testing what it seeks to, or it misses important aspects of job skills.

Would you want to fly in a plane, where the pilot knows how to take off but not land? Obviously not! Assessments for airline pilots take account all job functions including landing in emergency scenarios.

Similarly, if you are testing your employees to ensure competence for regulatory compliance purposes, or before you let them sell your products, you need to ensure the tests have content validity – that is to say they cover the job skills required.

Additionally to these common sense reasons, if you use an assessment without content validity to make decisions about people, you could face a lawsuit. See this blog post which describes a US lawsuit where a court ruled that because a policing test didn’t match the job skills, it couldn’t be used fairly for promotion purposes.

How can you increase content validity?

Here are some tips to get you started. For a deeper dive, Questionmark has several white papers that will help, and I also recommend Shrock & Coscarelli’s excellent book “Criterion-Referenced Test Development”.

  1. Conduct a job task analysis (JTA). A JTA is a survey which asks experts in the job role what tasks are important and how often they are done. A JTA gives you the information to define assessment topics in terms of what the job needs. Questionmark has a JTA question type which makes it easy to deliver and report on JTAs.
  2. Define the topics in the test before authoring. Use an item bank to store questions, and define the topics carefully before you start writing the questions. See Know what your questions are about before you deliver the test for some more reasoning on this.
  3. You can poll subject matter experts to check content validity for an existing test. If you have an existing assessment, and you need to check its content validity, get a panel of SMEs (experts) to rate each question as to whether it is  “essential,” “useful, but not essential,” or “not necessary” to the performance of what is being measured. The more SMEs who agree that items are essential, the higher the content validity. See Understanding Assessment Validity- Content Validity for a way to do this within Questionmark software.
  4. Use item analysis reporting. Item analysis reports flag questions which are don’t correlate well with the rest of the assessment. Questionmark has an easy to understand item analysis report which will flag potential questions for review. One of the reasons a question might get flagged is because participants who do well on other questions don’t do well on this question – this could indicate the question lacks content validity.
  5. Involve Subject Matter Experts (SMEs). It might sound obvious, but the more you involve SMEs in your assessment development, the more content validity you are likely to get. Use an assessment management system which is easy for busy SMEs to use, and involve SMEs in writing and reviewing questions.
  6. Review and update tests frequently. Skills required for jobs change quickly with changing technology and changing regulations.  Many workplace tests that were valid two years ago, are not valid today. Use an item bank with a search facility to manage your questions, and review and update or retire questions that are no longer relevant.

I hope this blog post reminds you why content validity matters and gives helpful tips to improve the content validity of your tests. If you are using a Learning Management System to create and deliver assessments, you may struggle to obtain and demonstrate content validity. If you want to see how Questionmark software can help manage your assessments, request a personalized demo today.

 

Questionmark is ISO 27001 certified. What this means for you?

Posted by John Kleeman

As you may have seen on our news site, Questionmark has just been certified to the Information Security standard ISO 27001.

What is ISO 27001?Chart showing probability and impact of risks, with high impact and high probability risks in red

ISO 27001 (full name ISO/IEC 27001:2013) is an international information security standard that is widely recognized as credible and authentic in validating that the certified organization has an effective management system for security.

The core of ISO 27001 is risk management. You identify in a systematic way risks to confidentiality, integrity, and availability and then assess their impact and probability. As simplistically shown in the diagram to the right, you decide what risks you can accept and how you can mitigate or otherwise deal with those that you cannot accept.

Subject to risk assessment, ISO 27001 requires you to meet over 100 controls including all the usually expected security controls. It also requires top management commitment and very specific processes to deal with issues that arise and auditing and much more.

ISO 27001 also encourages continual improvement – with all the threats out there, you have to keep making your processes and security better.

Questionmark’s ISO 27001 journey

Security has been central to Questionmark’s mission for decades. We brought out the world’s first secure browser in the 1990s, and our very first post in this blog back in 2009 was about delivering assessments safely and securely.

Last year, we decided that if we were to get external audit and validation of our security, it would both help us become more secure and help customers and other stakeholders feel more comfortable with our service. We’d been aware of 27001 for some time as the most credible security standard out there, and decided to adapt our processes and internal documentation to meet it. And we commissioned BSI, who are leaders in this field, to audit us.

The process to become certified is quite arduous. Including “internal” audits by a consultant and BSI’s audits, we have had eight days of auditing in the last few months. And these can be quite grueling – one of our audit days started with breakfast at 7 am and the auditor left the building just after 7.30pm at night! This definitely puts your people, processes, and technology through their paces. Implementing 27001 has improved Questionmark security and I’d encourage you to respect any organization who is certified as it’s a very credible process.BSI Assurance Mark Template RGB

I’m pleased to let you know that we are now certified by BSI under ISO 27001. Our certificate number is IS 668255. Our scope and certificate of applicability are wide, and we’d be pleased to share these with stakeholders under NDA.

How might it matter to purchasers of assessment services?

ISO 27001 certification gives external validation that an organization has a good quality information security management system.

Anyone can claim to be secure. Anyone can claim to follow standards. It’s hard for someone who is not a security expert to know whether an organization actually has put the effort into people, process, and technology to do the best that can be done to resist threats to confidentiality, integrity and availability.

With all the threats out there to assessment data, we believe it’s helpful to our customers to have assurance that Questionmark has been independently audited and it has been certified that our information security management system complies with ISO 27001.

How could ISO 27001 help assessment providers?

Are you looking to create and deliver secure assessments and keen to protect confidentiality, integrity, and availability?

Although using Questionmark OnDemand will help you do this, I’d encourage some blog readers to think whether it might make sense to implement ISO 27001 yourselves as an organization. That way you will ensure that all your IT and systems are securely managed. There are some work and effort involved, but it will make you as an organization more secure and less likely to suffer breaches and other failures.

I’ve just led Questionmark’s implementation of ISO 27001 and would be happy to share experiences with others in the assessment industry, please feel free to reach out to me.

Seven tips to recruit and manage SMEs for technology certification exams

imagePosted by John Kleeman

[repost from February 8, 2017]

How do you keep a certification exam up to date when the technology it is assessing is changing rapidly?

Certifications in new technologies like software-as-a-service and cloud solutions have some specific challenges. The nature of the technology usually means that questions often require very specialist knowledge to author. And because knowledge of the new technology is in short supply, subject matter experts (SMEs) who are able to author and review new items will be in high demand within the organization for other purposes.

Cloud technological offerings also change rapidly. It used to be that new technology releases came out every year or two, and if you were writing certification exams or other assessments to test knowledge and skill in them, you had plenty of notice and could plan an update cycle. But nowadays most technology organizations adopt an agile approach to development with the motto “release early, release often”. The use of cloud technology makes frequent, evolutionary releases – often monthly or quarterly – normal.

So how can you keep an exam valid and reliable if the content you are assessing is changing rapidly?

Here are seven tips that could help – a few inspired by an excellent presentation by Cisco and Microsoft at the European Association of Test Publishers conference.

  1. Try to obtain item writing SMEs from product development. They will know what is coming and what is changing and will be in a good position to write accurate questions. 
  2. Also network for SMEs outside the organization – at technology conferences, via partners and resellers, on social media and/or via an online form on your certification website. A good source of SMEs will be existing certified people.
  3. Incentivize SMEs – what will work best for you will depend on your organization, but you can consider free re-certifications, vouchers, discounts off conferences, books and other incentives. Remember also that for many people working in technology, recognition and appreciation are as important as financial incentives. Appreciate and recognize your SMEs. For internal SMEs, send thank you letters to their managers to appreciate their effort.
  4. Focus your exam on underlying key knowledge and skills that are not going to become obsolete quickly. Work with your experts to avoid items that are likely to become obsolete and seek to test on fundamental concepts, not version specific features.
  5. When working with item writers, don’t be frightened to develop questions based on beta or planned functionality, but always do a check before questions go live in case the planned functionality hasn’t been released yet.
  6. Analyze, create, deliverSince your item writers will likely be geographically spread and will be busy and tech-literate, use a good collaborative tool for item writing and item banking that allows easy online review and tracking of changes. (See https://www.questionmark.com/content/distributed-authoring-and-item-management for information on Questionmark’s authoring solution.)
  7. In technology as in other areas, confidentiality and exam security are crucial to ensure the integrity of the exam. You should have a formal agreement with internal and external SMEs who author or review questions to remind them not to pass the questions to others. Ensure that your HR or legal department are involved in the drafting of these so that they are enforceable.

Certification of new technologies helps adoption and deployment and contributes to all stakeholders success. I hope these tips help you improve your assessment program.

GDPR is coming. Are you ready?

Posted by Julie Delazyn

Don’t get left behind as the most important change in data privacy takes effect May 2018. The new General Data Protection Regulation (GDPR) intends to strengthen and unify privacy and data protection and any organization that stores or manages data about Europeans will need to comply.

With eye-watering regulatory fines of up to €20 million or 4% of global annual turnover (whichever is greater), a credible compliance strategy is essential.

Join us for a FREE 45 minute Webinar July 26, 2017, to understand how online assessments can help you meet your GDPR challenges.

The webinar will cover:

  • What the GDPR is and who it impacts
  • Why you should care about GDPR compliance
  • How to overcome the challenges presented by GDPR — including the learning curve for your employees
  • How assessment can help mitigate GDPR risks and aid your compliance strategy
  • Considerations for implementing assessment management software to aid in compliance

We look forward to speaking to you at the webinar!

Can you be GDPR compliant without testing your employees?

Posted by John Kleeman

The GDPR is a new extra-territorial, data protection law which imposes obligations on anyone who processes personal data on European residents. It impacts companies with employees in Europe, awarding bodies and test publishers who test candidates in Europe, universities and colleges with students in Europe and many others. Many North American and other non-European organizations will need to comply.

See my earlier post How to use assessments for GDPR compliance for an introduction to GDPR. The question this blog post addresses is whether it’s practical for a large organization to be compliant with the GDPR without giving tests and assessments to their employees?

I’d argue that for most organizations with 100s or 1000s of employees, you will need to test your employees on your policies and procedures for data protection and the GDPR. Putting it simply, if you don’t and your people make mistakes, fines are likely to be higher.

Here are four things the GDPR law says (I’ve paraphrased the language and linked to the full text for those interested):


1. Organizations must take steps to ensure that everyone who works for them only processes personal data based on proper instructions. (Article 32.4)

2. Organizations must conduct awareness-raising and training of staff who process personal data (Article 39.1). This is extended to include “monitoring training” for some organizations in Article 47.2.

3. Organizations must put in place risk-based security measures to ensure confidentiality and integrity and must regularly test, assess and evaluate the effectiveness of these measures. (Article 32.1)

4. If you don’t follow the rules, you could be fined up to 20 million Euros or 4% of turnover. How well you’ve implemented the measures in article 32 (i.e. including those above) will impact how big these fines might be. (Article 83.2d)


So let’s join up the dots.

Firstly, a large company has to ensure that everyone who works for it only processes data based on proper instructions. Since the nature of personal data, processing and instructions each have particular meanings, this needs training to help people understand. You could just train and not test, but given that the concepts are not simple, it would seem sensible to test or otherwise check their understanding.

A company is required to train its employees under Article 39. But the requirement in Article 32 is for most companies stronger. For most large organizations the risk of employees making mistakes and the risk of insider threat to confidentiality and integrity is considerable. So you have to put in place training and other security measures to reduce this risk. Given that you have to regularly assess and evaluate the effectiveness of these measures, it seems hard to envisage an efficient way of doing this without testing your personnel. Delivering regular online tests or quizzes to your employees is the obvious way to check that training has been effective and your people know, understand and can apply your processes and procedures.

Lastly, imagine your company makes a mistake and one of your employees causes a breach of personal data or commits another infraction under the GDPR? How are you going to show that you took all the steps you could to minimize the risk? An obvious question is whether you did your best to train that employee in good practice and in your processes and procedures? If you didn’t train, it’s hard to argue that you took the proper steps to be compliant. But even if you trained, a regulator will ask you how you are evaluating the effectiveness of your training. As a regulator in another context has stated:

“”where staff understanding has not been tested, it is hard for firms to judge how well the relevant training has been absorbed”

So yes, you can imagine a way in which a large company might manage to be compliant with the GDPR without testing employees. There are other ways of checking understanding, for example 1:1 interviews, but they are very time consuming and hard to roll out in time for May 2018. Or you may be lucky and have personnel who don’t make mistakes! But for most of us, testing our employees on knowledge of our processes and procedures under the GDPR will be wise.

Questionmark OnDemand is a trustable, easy to use and easy to deploy system for creating and delivering compliance tests and assessments to your personnel. For more information on using assessments to help ensure GDPR compliance visit this page of our website or register for our upcoming webinar on 29 June.

Next Page »