Twelve tips to make questions translation ready

Posted by John Kleeman

We all know the perils of mis-translation. My favourite mis-translation is the perhaps apocryphal tale of a laundry in Rome, Italy putting up a sign in English saying “Ladies, leave your clothes here and spend the afternoon having a good time.” With Questionmark having a translation management system to help you translate questions and assessments, here are some good practice tips on writing questions so they will be easy to translate.

1. Avoid questions that assume syntax is the same in all languages, for example fill-in-blank questions that rely on word order. For example, in English, the verb goes in the middle of a sentence but in Turkish and Korean, the verb is usually at the end of a sentence.

2. Also avoid “broken stem questions”, where the stem is an incomplete sentence and the participant must select the most appropriate answer to finish the sentence. That’s likely to be challenging to translate in some languages where the ordering may not make sense.

3. Keep questions simple. Avoid unnecessarily complex text or question stems with redundancy or unnecessary repetition. Such questions are best simplified before you translate them.

4. Avoid metaphors and idiomatic language in general; things like “in small steps” or “disappear into thin air”, could well introduce translation mistakes.

5. Avoid passive voice where you can. Not all languages make it easy to translate this, and it’s usually best to just use active voice.

6. Thoroughly review questions prior to translation to ensure no ambiguity. if the question wording is ambiguous, the translator’s interpretation of the question may not be the same as that of the question author.

7. If you are using a rating scale across many questions, investigate its cultural appropriateness and, if possible, whether it is widely used in the target language.

8. Test items based on nuances of vocabulary, descriptions of emotions or abstract concepts can be hard to translate, as different languages may have different vocabulary connotations.

9. You also need to be aware of the risk that translating a question could cause a question to become obvious due to different words in the target language, like the following Swedish example.

Question that shows a Swedish translation giving the answer to a question by words being the same

10. Avoid using cultural context within question stimulus. If you are presenting a scenario, make it one that is relevant to different cultures and languages. If it is difficult to avoid a culturally marked context, consider preparing good guidelines for translators in which you define what adaptations are encouraged, desirable and ruled out

11. If your question contains a graphic or video, consider if you can remove any text from it and still keep the question meaningful. Otherwise you need to translate the text in the graphic or video in each language.

12. if you are translating items into several languages, it can be cost effective to conduct a translatability assessment on the items before you do the detailed translation. This will alert to possible issues within various language families prior to doing the more substantial work of full translation. A translatability assessment lets you identify and fix issues early and relatively cheaply. See here for a blog from Steve Dept of CApStAn that explains more.

Thanks to Steve Dept for inspiring this blog post with an excellent conference presentation at EATP last year and for helping me write this article. For some more advice on translating and adapting tests, see the International Test Commission Guidelines for Translating and Adapting Tests or the Cross-cultural Survey Guidelines (CCSG), both of which have been recently updated.

I hope this advice helps you be efficient in your translation efforts. For information on Questionmark OnDemand which includes translation management system capabilities, see www.questionmark.com.

Trust Center: Our Commitment to Security

Posted by Julie Delazyn

Our commitment to security is real. Security and reliability are key to both to our success and the success of our customers. That’s why we’ve established our Trust Center — a single place us to feature our different accreditations and validations. In order to highlight Questionmark’s commitment to security, I spoke to Questionmark’s Executive Director and Founder John Kleeman about what security means to Questionmark and how far we go to ensure the protection and privacy of our customer’s data.

What does security mean to Questionmark?

Organizations and companies around the world — companies, universities, and colleges, certification organizations — entrust us with a large amount of confidential data from assessments. We conduct millions of assessments a year and customers depend on us to keep those secure. To Questionmark, security means: Confidentiality, Integrity and Availability of our services for our customers. It’s one of our central aims and goals.

How does Questionmark make security a priority?

You can never 100% ensure security. As you can see in the media, Governments or large well-regarded corporations have had security breaches –so it’s about reducing that risk and making it unlikely that there will be a security issue. The main way in which we do that is by having an information security management system and putting processes in place to look at the risks and put in place controls and take other actions to reduce them.

Anyone can claim to be secure. What sets us apart from some companies in the space is that we don’t just say we’re secure, we get external validation of our security. We’ve recently certified against the ISO 27001 standard, and that’s an involved, grueling audit, which looks at our procedures. By having external validation, we can prove that experts have looked at our security and checked that it’s real — not just that we’re claiming it.

Questionmark is a global company with customers around the world – how do the standards we meet reflect that?

Everyone in the world wants confidentiality, integrity and availability, but there are local compliance criteria—technical rules – that can differ. While ISO 27001 is an international accreditation, we’re also looking to meet various national standards. We’re in the process of establishing compliance with FedRAMP, which is a US Government requirement. We are an approved supplier under GSA in the US and the G-Cloud in the UK, we’ve also passed a Cyber Essentials certification in the UK, and we’re looking to pass other national and international accreditations. In Europe, the GDPR is a focus for European customers, and we’re making sure that we’re complaint with the GDPR. That compliance will help our customers around the world ensure that we follow best practices in data protection. But the main security issues are essentially the same in every country in the world and every geography.

What tools do we offer our customers to help them protect their own data?

The Questionmark software has a strong authentication structure in place—you can define password policies that allow you to have strong passwords. We support integration with SAML, which many of our customers use with their own systems. And then Questionmark has very flexible capabilities to give differential access to data. You can set up your system so only those people who need to know have access to data in your Questionmark system. These are the kinds of features in the product that genuinely increase security.

Why the trust center?

We know with all the concerns about cyber security that there is a real need for our customers to understand whether we as a supplier are secure and safe for them, and so we wanted to create a single place to feature these different accreditations and validations.

There are only a few ways to prove you are secure. Let’s take the example of a castle—you or I could look at this massive structure and guess that if it’s secure or not. But if we’re not castle experts, we really can’t be sure. Similarly, unless you’re a software as a service security expert, how do you know if company A, B or C is secure? By getting to see the different certifications and laws that we follow, customers can get an appreciation of the security we offer, and compare that to others.

What is the future for Questionmark’s commitment to security?

You must have continued improvement. Threats are evolving, and you just can’t stand still. As a very simple example, we currently test our own employees on data security every year, and we’re expanding that by planning different tests by role or department. We’re looking at other accreditations and a continual increase in technical controls. We have many internal improvements that we’re expanding to implement soon. It’s literally a process of continual improvement, and we will continue to add to the trust center.

Questionmark is committed to the highest levels of trust, transparency, and compliance. Please register for a free intro webinar for more information on our key features and functions. We look forward to building your trust and working with you to secure your data.

6 Stages to Getting Assessment Results You Can Trust: White Paper

Headshot JuliePosted by Julie Delazyn

Modern organizations need their people to be competent.

Would you be comfortable in a high-rise building designed by an unqualified architect? Would you fly in a plane whose pilot hadn’t passed a flying test? Would you send a sales person out on a call  if they didn’t know what your products do? Can you demonstrate to a regulatory authority that your staff are competent and fit for their jobs if you do not have trustworthy assessments?

In all these cases and many more, it’s essential to have a reliable and valid test of competence. If you do not ensure that your workforce is qualified and competent, then you should not be surprised if your employees have accidents, cause your organization to be fined for regulatory infractions, give poor customer service or can’t repair systems effectively.

The white paper, Assessment Results you Can Trust, explains that trustworthy assessment results must be both valid (measuring what you are looking for them to measure) and reliable (consistently measuring what you want to be measured).

For assessments to be valid and reliable, it’s necessary to follow structured processes at each step from planning through authoring to delivery and reporting.

The white paper covers these six stages of the assessment process:

  • Planning assessment
  • Authoring items
  • Assembling assessment
  • Pilot and review
  • Delivery
  • Analyze results

Following the advice in the white paper and using the capabilities it describes will help you produce assessments that are more valid and reliable — and hence more trustable.

To download the complimentary white paper, click here.


How do you improve exam integrity and candidate satisfaction?

Posted by Julie Delazyn

Proxy testing … Cheating …. Content theft. How can you be sure that the integrity of your exam isn’t compromised when the technologies and techniques are complex and continually evolving? Many organizations are increasingly turning to online proctoring solutions.

Research shows that when properly applied, online proctoring can be as or more effective than traditional test-center proctoring. Besides reducing security risks, it also improves candidate satisfaction by offering the convenience of taking tests from their homes, workplaces or colleges. It also reduces costs not only for administrators but also for test takers. Consider for a moment the testing fees, travel, parking, and the simple time away from work or studies.  In addition to online proctoring’s effectiveness, one of the most exciting things is that this approach has the potential to grow as technology becomes more advanced and widespread.

What do you need to consider before implementing online proctoring? What features are important to you? How will your exams retain their integrity? How are other colleges or credentialing organizations using online proctoring to improve candidate experience?

Join Questionmark for a 30-minute webinar answering these questions and more

The webinar will cover:

  • Record and review based system with automated flagging
  • What colleges, universities and credentialing organizations are using to offer 24/7 flexibility to test-takers
  • How to administer online exams in a fair way by combining test delivery with exam proctoring

Want to learn more? Secure your spot for this complimentary webinar

Six tips to increase content validity in competence tests and exams

Posted by John Kleeman

Content validity is one of the most important criteria on which to judge a test, exam or quiz. This blog post explains what content validity is, why it matters and how to increase it when using competence tests and exams within regulatory compliance and other work settings.

What is content validity?

An assessment has content validity if the content of the assessment matches what is being measured, i.e. it reflects the knowledge/skills required to do a job or demonstrate that the participant grasps course content sufficiently.

Content validity is often measured by having a group of subject matter experts (SMEs) verify that the test measures what it is supposed to measure.

Why does content validity matter?

If an assessment doesn’t have content validity, then the test isn’t actually testing what it seeks to, or it misses important aspects of job skills.

Would you want to fly in a plane, where the pilot knows how to take off but not land? Obviously not! Assessments for airline pilots take account all job functions including landing in emergency scenarios.

Similarly, if you are testing your employees to ensure competence for regulatory compliance purposes, or before you let them sell your products, you need to ensure the tests have content validity – that is to say they cover the job skills required.

Additionally to these common sense reasons, if you use an assessment without content validity to make decisions about people, you could face a lawsuit. See this blog post which describes a US lawsuit where a court ruled that because a policing test didn’t match the job skills, it couldn’t be used fairly for promotion purposes.

How can you increase content validity?

Here are some tips to get you started. For a deeper dive, Questionmark has several white papers that will help, and I also recommend Shrock & Coscarelli’s excellent book “Criterion-Referenced Test Development”.

  1. Conduct a job task analysis (JTA). A JTA is a survey which asks experts in the job role what tasks are important and how often they are done. A JTA gives you the information to define assessment topics in terms of what the job needs. Questionmark has a JTA question type which makes it easy to deliver and report on JTAs.
  2. Define the topics in the test before authoring. Use an item bank to store questions, and define the topics carefully before you start writing the questions. See Know what your questions are about before you deliver the test for some more reasoning on this.
  3. You can poll subject matter experts to check content validity for an existing test. If you have an existing assessment, and you need to check its content validity, get a panel of SMEs (experts) to rate each question as to whether it is  “essential,” “useful, but not essential,” or “not necessary” to the performance of what is being measured. The more SMEs who agree that items are essential, the higher the content validity. See Understanding Assessment Validity- Content Validity for a way to do this within Questionmark software.
  4. Use item analysis reporting. Item analysis reports flag questions which are don’t correlate well with the rest of the assessment. Questionmark has an easy to understand item analysis report which will flag potential questions for review. One of the reasons a question might get flagged is because participants who do well on other questions don’t do well on this question – this could indicate the question lacks content validity.
  5. Involve Subject Matter Experts (SMEs). It might sound obvious, but the more you involve SMEs in your assessment development, the more content validity you are likely to get. Use an assessment management system which is easy for busy SMEs to use, and involve SMEs in writing and reviewing questions.
  6. Review and update tests frequently. Skills required for jobs change quickly with changing technology and changing regulations.  Many workplace tests that were valid two years ago, are not valid today. Use an item bank with a search facility to manage your questions, and review and update or retire questions that are no longer relevant.

I hope this blog post reminds you why content validity matters and gives helpful tips to improve the content validity of your tests. If you are using a Learning Management System to create and deliver assessments, you may struggle to obtain and demonstrate content validity. If you want to see how Questionmark software can help manage your assessments, request a personalized demo today.

 

Questionmark is ISO 27001 certified. What this means for you?

Posted by John Kleeman

As you may have seen on our news site, Questionmark has just been certified to the Information Security standard ISO 27001.

What is ISO 27001?Chart showing probability and impact of risks, with high impact and high probability risks in red

ISO 27001 (full name ISO/IEC 27001:2013) is an international information security standard that is widely recognized as credible and authentic in validating that the certified organization has an effective management system for security.

The core of ISO 27001 is risk management. You identify in a systematic way risks to confidentiality, integrity, and availability and then assess their impact and probability. As simplistically shown in the diagram to the right, you decide what risks you can accept and how you can mitigate or otherwise deal with those that you cannot accept.

Subject to risk assessment, ISO 27001 requires you to meet over 100 controls including all the usually expected security controls. It also requires top management commitment and very specific processes to deal with issues that arise and auditing and much more.

ISO 27001 also encourages continual improvement – with all the threats out there, you have to keep making your processes and security better.

Questionmark’s ISO 27001 journey

Security has been central to Questionmark’s mission for decades. We brought out the world’s first secure browser in the 1990s, and our very first post in this blog back in 2009 was about delivering assessments safely and securely.

Last year, we decided that if we were to get external audit and validation of our security, it would both help us become more secure and help customers and other stakeholders feel more comfortable with our service. We’d been aware of 27001 for some time as the most credible security standard out there, and decided to adapt our processes and internal documentation to meet it. And we commissioned BSI, who are leaders in this field, to audit us.

The process to become certified is quite arduous. Including “internal” audits by a consultant and BSI’s audits, we have had eight days of auditing in the last few months. And these can be quite grueling – one of our audit days started with breakfast at 7 am and the auditor left the building just after 7.30pm at night! This definitely puts your people, processes, and technology through their paces. Implementing 27001 has improved Questionmark security and I’d encourage you to respect any organization who is certified as it’s a very credible process.BSI Assurance Mark Template RGB

I’m pleased to let you know that we are now certified by BSI under ISO 27001. Our certificate number is IS 668255. Our scope and certificate of applicability are wide, and we’d be pleased to share these with stakeholders under NDA.

How might it matter to purchasers of assessment services?

ISO 27001 certification gives external validation that an organization has a good quality information security management system.

Anyone can claim to be secure. Anyone can claim to follow standards. It’s hard for someone who is not a security expert to know whether an organization actually has put the effort into people, process, and technology to do the best that can be done to resist threats to confidentiality, integrity and availability.

With all the threats out there to assessment data, we believe it’s helpful to our customers to have assurance that Questionmark has been independently audited and it has been certified that our information security management system complies with ISO 27001.

How could ISO 27001 help assessment providers?

Are you looking to create and deliver secure assessments and keen to protect confidentiality, integrity, and availability?

Although using Questionmark OnDemand will help you do this, I’d encourage some blog readers to think whether it might make sense to implement ISO 27001 yourselves as an organization. That way you will ensure that all your IT and systems are securely managed. There are some work and effort involved, but it will make you as an organization more secure and less likely to suffer breaches and other failures.

I’ve just led Questionmark’s implementation of ISO 27001 and would be happy to share experiences with others in the assessment industry, please feel free to reach out to me.

Next Page »