Ten Tips to Translate Tests Thoughtfully

John KleemanPosted by John Kleeman

Tests and exams are used for serious purposes and have significant impact on people’s lives. If they are translated wrongly, it can result in distress. As a topical illustration, poor translation of an important medical admissions test in India was the subject of a major law case ruled on by the Indian Supreme Court last week.

Because language and cultures vary, fairly translating tests and exams is hard. I recently attended a seminar organized by the OECD on translating large scale assessments which gave me a lot of insight into the test translation process.  If you are interested  in the OECD seminar, Steve Dept of Questionmark partner cApStAn has written a blog here, and the seminar presentations are available on the OECD website.

Here are some tips from what I’ve learned at the seminar and elsewhere on good practice in translating tests and exams.

  1. Put together a capable translation management team. A team approach works well when translating tests. For example a subject matter expert, a linguist/translator, a business person and a testing expert would work well together as a review and management committee.
  2. Think through the purpose of your translation. Experts say that achieving perfect equivalence of a test in two languages is close to impossible, so you need to define your goals. For example, are you seeking to adapt the test to measure the same thing or are you looking for a literal translation? The former may be more realistic especially if your test includes some culturally specific examples or context.  Usually what you will be looking for is that the test in two languages is comparable in that a pass score in the test in either language means a similar thing for competence.
  3. Define a glossary for your project. If your test is on a specialist or technical subject, it will have some words specific to the content area. You can save time and increase the quality of the translation if you identify the expected translation of these words in advance. This will guide the translating team and ensure that test takers see consistent vocabulary.
  4. Use a competent translator (or translation company). A translator must be native in the target language but also needs current cultural knowledge, ideally from living in the target locale. A translator who is not native to the language will not be effective, and a translator who does not have knowledge of the culture may miss some references in question content  (e.g. local names or slang). An ideal translator will also have subject matter knowledge and assessment knowledge.
  5. Diagram showing export into XLIFF XML and then re-importExport to allow a translator to use their own tools. Translators have many automated tools available to them including translation memories, glossaries and automated checking systems. For simple translation, you can translate interactively within an assessment system, but you will get more professional results if you export from your assessment management system, allow the translator to translate in their system, and then re-import (as shown in the diagram).
  6. Put in place a verification procedure. Translators are human and make mistakes, questions can also rely on context or knowledge that a translator may not have. A verification process will involve manual review by stakeholders looking at things like accuracy, style, country issues, culture, no clues given in choices, right choice not obviously longer than other choices and different translation word choices used in stem/choices.
  7. Also review by piloting and looking at item difficulty. Linguistic review is helpful but you should also look at item performance in practice. The difficulty of a translated item will vary slightly between languages. Generally small errors will be up and down and roughly cancel out. You want to catch the big errors, where ambiguity or mis-translation makes a material difference to test accuracy. You can catch some of these by running a small pilot to 50 (or even 25) participants and comparing the p-value (item difficulty or proportion who get right) in the languages. This can flag questions with significant differences in difficulty; such questions need review as they may well be badly translated.
  8. Consider using bilingual reviewers. If you have access to bilingual people (who speak the target and source language), it can be worth asking them to look at both versions of the questions and comment. This shouldn’t be your only verification procedure but can be very helpful and spot issues.
  9. Update translations as questions change. In any real world test, questions in your item bank get updated over time, and that means you need to update the translations and keep track of which ones have been updated in which languages. It can be helpful  to use a translation management system, for example the one included within Questionmark OnDemand to help you manage this process, as it’s challenging and error-prone to manage manually.
  10. Read community guidelines. The International Test Commission have produced well-regarded guidelines on adapting/ translating tests – you can access them here. The OECD PISA guidelines, although specific to the international PISA tests, have  good practice applicable to other programs. I personally like the heading to one of the sections in the PISA guidance: “Keep in mind that some respondents will misunderstand anything that can be misunderstood”!

I hope you found this post interesting – all suggestions are personal and not validated by the OECD or others. If you did find it interesting, you may also want to read my earlier blog post: Twelve tips to make questions translation ready.

To learn more about Questionmark OnDemand and Questionmark’s translation management system, see here or request a demo.

Translatability in Questionmark Perception Version 5

john_smallPosted by John Kleeman

Making assessments and questions translatable was a key goal for Questionmark Perception version 5.

There always has been a need to translate assessments, particularly in countries like Switzerland and Canada, which have have different languages within their borders. But the Internet has made the world much more connected, and so many organizations have employees and stakeholders in different countries or speaking different languages. Some typical translation needs that fed into version 5 were:

  • A bank that operates in many parts of the world and wants to deliver course evaluation surveys (level 1s) to participants in training in many languages. It’s crucial for them that they can schedule the assessment and allow their participants to choose the language in which the assessment is taken
  • A telephone company that trains employees in Europe and wants to give similar questions to all employees. They want to author the questions once and then translate them for use by training teams in each country.
  • A university that creates and delivers questions in English but is expanding internationally, is in partnership with universities overseas and wants to translate some of its material into other languages
  • A software company that deliver certifications to consultants and partners worldwide in around 20 languages and needs assessments and questions translated so that the certifications can be given fairly to anyone who speaks the languages they support.
  • A manufacturing support company that authors questions in English and then translates them into 8 European and Asian languages for delivery to their partners and employees worldwide.

Some customers use external translators and want Questionmark to export XML that translators can use in specialist tools, whilst other customers have internal expertise and want to translate in house and need screens within Questionmark Perception to make the translations themselves. For all customers, it’s not just the initial translation that they need help with but the management of the process. When questions change, if you have them translated into a dozen language, it’s a nightmare to keep track manually of what needs updated, and Questionmark software needs to flag when a question has changed and remind you to update the translation.

The basic concept of Questionmark’s translation management system is that you create a question in a base language and then translate it into as many target languages as you want. And then you do the same with an assessment.

translating501g

Questions and assessments are linked so that you can report across language.

And to do the translation, you can translate in Perception by typing in the text in a simple user interface shown below.

image

Translating within Questionmark Perception works well if you have an in-house translator, but if you are working with external translators, it’s usually best to export to XML and send them the text to translate – and they will return it to you to import into Perception. Text is exported in an industry standard XML called XLIFF that standard software packages used by translators can process.

Whether you translate interactively or by export to XML, Questionmark keeps track of when questions change in the base language and prompts you to update translations to keep translated questions up to date. So we help you not just manage the initial translation process, but also the ongoing process of translations as questions and assessments change and evolve.

We’re very excited about how easy it is to translate questions in Perception version 5, and we look forward to your feedback as you use it to create quizzes, tests, exams and surveys that can be used in many languages.