Ten Tips to Translate Tests Thoughtfully

John KleemanPosted by John Kleeman

Tests and exams are used for serious purposes and have significant impact on people’s lives. If they are translated wrongly, it can result in distress. As a topical illustration, poor translation of an important medical admissions test in India was the subject of a major law case ruled on by the Indian Supreme Court last week.

Because language and cultures vary, fairly translating tests and exams is hard. I recently attended a seminar organized by the OECD on translating large scale assessments which gave me a lot of insight into the test translation process.  If you are interested  in the OECD seminar, Steve Dept of Questionmark partner cApStAn has written a blog here, and the seminar presentations are available on the OECD website.

Here are some tips from what I’ve learned at the seminar and elsewhere on good practice in translating tests and exams.

  1. Put together a capable translation management team. A team approach works well when translating tests. For example a subject matter expert, a linguist/translator, a business person and a testing expert would work well together as a review and management committee.
  2. Think through the purpose of your translation. Experts say that achieving perfect equivalence of a test in two languages is close to impossible, so you need to define your goals. For example, are you seeking to adapt the test to measure the same thing or are you looking for a literal translation? The former may be more realistic especially if your test includes some culturally specific examples or context.  Usually what you will be looking for is that the test in two languages is comparable in that a pass score in the test in either language means a similar thing for competence.
  3. Define a glossary for your project. If your test is on a specialist or technical subject, it will have some words specific to the content area. You can save time and increase the quality of the translation if you identify the expected translation of these words in advance. This will guide the translating team and ensure that test takers see consistent vocabulary.
  4. Use a competent translator (or translation company). A translator must be native in the target language but also needs current cultural knowledge, ideally from living in the target locale. A translator who is not native to the language will not be effective, and a translator who does not have knowledge of the culture may miss some references in question content  (e.g. local names or slang). An ideal translator will also have subject matter knowledge and assessment knowledge.
  5. Diagram showing export into XLIFF XML and then re-importExport to allow a translator to use their own tools. Translators have many automated tools available to them including translation memories, glossaries and automated checking systems. For simple translation, you can translate interactively within an assessment system, but you will get more professional results if you export from your assessment management system, allow the translator to translate in their system, and then re-import (as shown in the diagram).
  6. Put in place a verification procedure. Translators are human and make mistakes, questions can also rely on context or knowledge that a translator may not have. A verification process will involve manual review by stakeholders looking at things like accuracy, style, country issues, culture, no clues given in choices, right choice not obviously longer than other choices and different translation word choices used in stem/choices.
  7. Also review by piloting and looking at item difficulty. Linguistic review is helpful but you should also look at item performance in practice. The difficulty of a translated item will vary slightly between languages. Generally small errors will be up and down and roughly cancel out. You want to catch the big errors, where ambiguity or mis-translation makes a material difference to test accuracy. You can catch some of these by running a small pilot to 50 (or even 25) participants and comparing the p-value (item difficulty or proportion who get right) in the languages. This can flag questions with significant differences in difficulty; such questions need review as they may well be badly translated.
  8. Consider using bilingual reviewers. If you have access to bilingual people (who speak the target and source language), it can be worth asking them to look at both versions of the questions and comment. This shouldn’t be your only verification procedure but can be very helpful and spot issues.
  9. Update translations as questions change. In any real world test, questions in your item bank get updated over time, and that means you need to update the translations and keep track of which ones have been updated in which languages. It can be helpful  to use a translation management system, for example the one included within Questionmark OnDemand to help you manage this process, as it’s challenging and error-prone to manage manually.
  10. Read community guidelines. The International Test Commission have produced well-regarded guidelines on adapting/ translating tests – you can access them here. The OECD PISA guidelines, although specific to the international PISA tests, have  good practice applicable to other programs. I personally like the heading to one of the sections in the PISA guidance: “Keep in mind that some respondents will misunderstand anything that can be misunderstood”!

I hope you found this post interesting – all suggestions are personal and not validated by the OECD or others. If you did find it interesting, you may also want to read my earlier blog post: Twelve tips to make questions translation ready.

To learn more about Questionmark OnDemand and Questionmark’s translation management system, see here or request a demo.

Web seminar: Assessment translation, localization and adaptation

Posted by Joan Phaup

“Please leave your values at the front desk.” This message, seen in a Paris hotel elevator, is among the many amusing translations that appear on signs and posters around the world.

We all chuckle over linguistic blunders like this one, but mistakes in translation, localization and adaptation are no laughing matter when delivering assessments to multilingual and multicultural audiences.

On Thursday, February 16, we will host a web seminar addressing this important subject: Assessment Translation, Localization and Adaptation: Expanding the Reach of your Testing Program

Sue Orchard

You can choose between two sessions:

The key presenter will be Sue Orchard, managing director and founder of Comms Multilingual, a professional translation services firm specializing in complex translation projects including test, assessment and exam content.

Sue will explain what it takes to ensure an assessment is adapted appropriately for target languages and cultures — something increasingly important as globalization continues to remove geographic barriers and open new economic opportunities. The session will also include some information about how Questionmark technologies can be used to aid translation management and multilingual assessment delivery.

I asked Sue for some details about her topic and what people can expect from the web seminar:

What will you cover during the presentation?

Translation, Localization and Adaptation is a bit of a mouthful so I will shorten it to TLA. In my presentation, I will be looking at the steps and processes which are necessary to ensure a successful TLA project. I will be examining some of the pitfalls and challenges with a TLA project and looking at ways to overcome these.

What are some of those pitfalls?

I think that the major issue is that TLA projects are usually an afterthought. People spend a lot of time and effort getting things right in the original language and then don’t allow enough time for the TLA element. Also, I think that it is very important that people think about future language versions when putting together the material in the original language.

Things such as text expansion, when materials are translated into languages such as Spanish and French, really need to be taken into account. French and Spanish can take up to 30% more space than an original English text, for example. Additionally, people need to think about how suitable their material is for translation. Are there a lot of items that can’t just be simply translated, but which must be adapted and localized first?

Is there one key piece of advice you give to organizations running multilingual assessments?

There are two key words that I would use: Preparation and Planning.

So many times the translation element of a project is an afterthought and everything has to be really rushed. This is when problems will occur. If you think about the amount of time it has taken to prepare and put together the original test, exam or assessment in the original language, then you can’t expect to have a successful translation project if you try to do things at the last minute. It is very important to allow enough time to ensure a successful outcome.

What do you hope your listeners will take away from your presentation?

I hope to give participants a guide to best practice when it comes to TLA projects. After my presentation, I hope that people will feel happier and more confident when it comes to translation projects and that they will know what needs to be done to ensure a successful TLA project.


For more details and free registration, choose one of links above