Limit item exposure with “Avoid Previously Delivered Questions”

Posted by Brian McNamara

When you’re allowing a participant multiple attempts on a quiz or test, you might want to be able to deliver ‘new’ questions to the participant each time that he or she takes the assessment. In other words, you might wish to ensure that the participant doesn’t get the same question on subsequent attempts — particularly if your assessment is drawing questions from a large item bank.

Questionmark authors can select “Avoid Previously Delivered Questions” when creating an assessment that draws questions at random from a particular topic.  This option is available when choosing items in the assessment editor or assessment wizard:

Once you select this option for a particular assessment, participants who have multiple attempts on the assessment will not be delivered the same item on subsequent attempts.

This feature can be useful for both quizzes designed to assist in the learning process as well as for medium and higher stakes assessments where administrators are looking to limit item exposure to candidates.

The Role of Assessments in Mitigating Risk for Financial Services Organizations

Posted by John Kleeman

Ever since Questionmark started in the 1980s, one of our largest areas of use has been the financial services industry.   Mitigating risk is key component of success for banks, insurance companies and others in financial services. Assessments play a crucial role here, and many of the world’s leading financial services companies use Questionmark assessments to help mitigate risk.

With colleagues, I’ve written a white paper on The Role of Assessments in Mitigating Risk for Financial Services Organizations. You can download the white paper, it’s free with registration. I’d like to thank several readers of this blog for their review and input into draft versions of the white paper.

Mitigating risk is key component of success in the financial services industry. The risk of non-compliance is an obvious concern, but other risks abound, too – such as losing customers and/or good employees.

Every organization defines processes to mitigate risk and maintain compliance, and it’s essential for employees to understand them and follow these processes. If they don’t, the risk of non-compliance increases. Training certainly helps, but without follow-up assessments there’s no proof that people have learned what they need to know. And organizations that can’t prove they have taken adequate steps to educate employees face higher non-compliance penalties.

How can organizations ensure that employees have demonstrated competence in matters of regulatory compliance?

Assessments play an essential role: Quizzes, tests and surveys are the most cost-effective way to confirm and document mandatory regulatory training. Whether required by regulators or not, assessments delivered online to employees are one of the few ways — and likely the best way — to touch individually on your entire workforce and ensure that people understand their roles in your business and what is required of them to meet regulatory and business needs. Effective assessments improve performance – making for happier customers and employees – while at the same time aiding compliance.

Centrally managed assessments delivered via IT technology bring both peace of mind and tremendous business value by:

  • helping employees develop the knowledge skills and attitudes to perform well
  • ensuring that employees act competently and follow proper procedures
  • demonstrating your organization’s commitment to complying with laws, making prosecution less likely if an individual missteps
  • providing the most conclusive and effective documentation that training has taken place and that employees understand what they are required to do
  • warning of a lack of knowledge or understanding before these impact the business
  • highlighting training needs so you can direct resources towards areas of weakness
  • saving time by allowing knowledgeable employees to “test out” of training on topics they are already expert in
  • allowing you to turn a compliance program that simply focuses on regulations into one that helps your employees improve customer service

Online assessments have long been used in compliance, but now more than ever. Most large companies today see online assessments as a critical part of their compliance programs. The paper explains why, by describing today’s regulatory landscape and showing the business benefits of using assessments to meet its demands. It also describes specific ways to use assessments within a compliance program, examines security and accessibility issues and offers role-specific best practice guidance for implementing a legally defensible assessment program.

I hope the paper proves useful reading!

How much do you know about assessment? Quiz 5: Designing Assessments

Posted by John Kleeman

Here is the fifth of our series of quizzes on assessment subjects, authored in conjunction with Neil Bachelor of Pure Questions. This week’s quiz is on designing assessments, a large topic area, but we’ve selected some interesting questions to check your understanding.

Here are links to earlier quizzes in the series:

Quiz 1 – Cut Scores
Quiz 2 – Validity & Defensibility
Quiz 3 – Use of formative quizzes
Quiz 4 – Trialling Questions

We regard resources like this quiz as a way of contributing to the ongoing process of learning about assessment. In that spirit, please enjoy the quiz below and feel free to comment if you have any suggestions to improve the questions.

Managing Assessment Translations

Posted By Doug Peterson

Joan Phaup recently published a blog article — “International Certifications: To Translate or Not to Translate?” – in which she interviewed Sue Orchard of Comms Multilingual about the need to translate assessments into different languages, and things to look out for when doing so. Even small organizations are realizing that they may need to translate their assessments into different languages because of the world-wide reach of the Internet or because the local community they serve is increasing in its ethnic diversity. The good news is that Questionmark’s Translation Management System can help you organize and maintain your translation work!

The Translation Management System is a licensable feature accessible through Enterprise Manager, on the Authoring tab. There are two main components to this system: projects and updates tracking. You can create a project around translating items, and the project will track into which languages the items have been translated. You can also specify various statuses for each item to help you track the items through a workflow involving translation, review and approval.

Once the items are translated, you can create an assessment project to translate an assessment made up of your translated items. Once the assessment is created in the base language, you don’t have to rebuild it for each new language – the Translation Management System will automatically build the translated version of the assessment for you using the appropriate language version of the constituent items. All you have to do is translate the text elements specific to the assessment itself (e.g., instructions, feedback, etc.) and Questionmark will take care of the rest.

This system also helps you track updates to your items. When you change the base-language version of an item or assessment, you will be able to see which translated versions need attention. This makes it easy to keep track of updates across multiple languages.

But what if you aren’t doing the translations yourself? For example, maybe you’ve hired an outside vendor to do your translation work for you. That’s no problem! You can export your translation project to an industry standard XLIFF file. (XLIFF stands for XML Localization Interchange File Format.) You can then send that file to your translator, let them do their work, and then import the updated file they send back right back into your project in Questionmark.

Once you have your translations in place, you have a couple of options when it comes to scheduling the assessment. You can specify the language in which the assessment should be delivered, or you can allow the participant to choose from the available translations.

I hope I’ve explained the basics of this feature, but here is a link to the Translation Management System page on our website, where you can read more on the subject and try it out for yourself.

If you’d like to see what it’s like to take a multilingual assessment, click here, choose from four different languages and take our Round the World quiz!

The Future Looks Bright

Posted by Jim Farrell

Snapshot from a “Future Solutions” focus group

Our Users Conferences are a time for us to celebrate our accomplishments and look forward to the challenges that lie in front of us. This year’s conference was full of amazing sessions presented by Questionmark staff and customers. Our software is being used to solve complex business problems. From a product standpoint, it is the very exciting to bring these real-life scenarios to our development teams to inspire them.

So where do we go from here? The Conference is our chance to stand in front of our customers and get feedback on our roadmap. We also held smaller “Future Solutions” focus groups to get feedback from our customers on what we have done and what we could do in the future to help them. In the best of times, these are an affirmation that we are on the right path. This was definitely one of those years.

One of our Future Solutions sessions focused on authoring. During that session, Doug Peterson and I laid out the future of Questionmark Live. This included an aggressive delivery cycle that will bring future releases at a rapid pace. Stay tuned for videos on new features available soon.

Ok…enough about us. This conference is really about our customers. The panel and peer discussion strand of this year’s conference had some of the most interesting topics. John Kleeman has already mentioned the security panel with our friends from Pearson Vue, ProctorU, Innovative Exams and Shenandoah University.

Another session that stood out was as a peer discussion test defensibility using the Angoff method to set cut scores. This conversation was very  interesting to me as someone who once had to create defensible assessments. I am eager to see organizations utilize Angoff because not only do  you want legally defensible assessments, you want to define levels of competency for a role and be able to determine how that can  predict future performance.

For those of you who do not know, the Angoff method is a way for Subject Matter Experts (SMEs) to grade the probability of a marginal student getting a question right. Attendees at this conference session were provided a handout that includes a seven-step flowchart guiding them in the design, development and implementation of the Angoff method.

If you are interested in Angoff and setting test scores I highly recommend reading Criterion-Referenced Test Development written by our good friends Sharon Shrock and Bill Coscarelli.

We really hope to see everyone at the 2013 Users Conference in Baltimore March 3 – 6. (I am hoping we may even get a chance to visit the beautiful Camden Yards!)