Q&A: High-stakes online tests for nurses

Headshot JuliePosted by Julie Delazyn

I spoke recently with Leanne Furby, Director of Testing Services at the National League for Nursing (NLN), about her case study presentation at the Questionmark 2015 Users Conference in Napa Valley March 10-13.

Leanne’s presentation, Transitioning 70 Years of High-Stakes Testing to Questionmark, explains NLN’s switch from a proprietary computer- and paper-based test delivery engine to Questionmark OnDemand for securely delivering standardized exams worldwide. I’m happy to share a snippet from of our conversation:

Tell me about the NLN

The NLN is a national organization for faculty nurses and leaders in nurse education. We offer faculty development, networking opportunities, testing services, nursing research grants and public policy initiatives to more than 26,000 members.

Why did you switch to Questionmark?

Our main concern was delivering our tests and exams to a variety of different devices. We wanted our students to be able to take a test on a tablet or take a quiz on their own mobile devices, and this wasn’t something we could do with our proprietary test delivery engine.

Our second major reason to go with Questionmark was the Customized Assessment Reports and the analytics tools. Before making the switch, we were having to create reports and analyze results manually. It took time and resources. Now this is all integrated in Questionmark.

How do you use Questionmark assessments?

We have 90 different exam lines and deliver approximately 75,000 to 100,000 secure exams a year, both nationally and internationally, in multiple languages. The NLN partnered with Questionmark in 2014 to transition the delivery of these exams through a custom-built portal. Questionmark is now NLN’s turnkey solution—from item banking and test development with SMEs all over the world to inventory control, test delivery and analytics.

This transition has had a positive outcomes for both our organization and our customers. We have developed a new project management policy, procedures for system transition and documentation for training at all levels. This has transformed the way we develop, deliver and analyze exams and the way we collect data for business and education purposes.

What are you looking forward to at the conference?

I am most looking forward to the opportunity to speak to other users and product developers to learn tips, tricks and little secrets surrounding the product. It’s so important to speak to people who have experience and can share ways of utilizing the software in ways you hadn’t thought of.

Thank you Leanne for taking time out of your busy schedule to discuss your session with us!

***

You have the opportunity to save $100 on your own conference registration: Just sign up by January 29 to receive this special early-bird discount.

What is the best way to reduce cheating?

John Kleeman HeadshotPosted by John Kleeman

There is a famous saying: “If you want to build a ship, don’t drum up the people to gather wood, divide the work, and give orders. Instead, teach them to yearn for the vast and endless sea.” This has a useful analogy in preventing cheating.

There are many useful technical and procedural ways of preventing cheating in tests and exams, and these are important to follow, but an additional, cost-effective way of reducing cheating is to encourage participants to choose not to cheat. If you can make your participants want to take the test fairly and honestly — by reducing their rationalization to cheat — this will reduce cheating.

Fraud triangle - Motivation, Opportunity and RationalizationAs shared by my colleague Eric Shepherd  in his excellent blog article Assessment Security and How To Reduce Fraud, cheating at a test is a variant of fraud.  Donald Cressey, a famed criminologist came up with the fraud triangle shown in the diagram to the right to explain why people commit fraud.

In order for someone to commit fraud (e.g. cheat at a test), he or she must have Motivation, Opportunity and Rationalization.  Motivation comes from the stakes of the test; for an important test, this is difficult to reduce. Opportunity arises out of technical and procedural weaknesses in the test-taking process, and you can obviously strengthen processes to reduce opportunity in many ways.

Rationalization is when someone reconciles their bad deeds as acceptable behavior. We all have values and like to think that what we are doing is right. When someone conducts fraud, they typically rationalize to themselves that what they are doing is right or at least acceptable. For example, they convince themselves that the organization they are robbing deserves it or can afford the loss. When cheating at a test, they say to themselves that the test is not fair or that they are just copying everyone else or they find some other excuse to rationalize and feel good about the cheating.

Here are some ways to make it less likely that people will rationalize about cheating on your test:

1. Formalize a code of conduct (e.g. honesty code) which sets out what you expect from test takers. Communicate this effectively well in advance and get people to sign up to it right before taking the test. For example, you can put it on the first screen after they log in. This will reduce rationalization from people who might claim to themselves they didn’t know it was wrong to cheat or that everyone cheats.

2. Provide fair and accessible learning environments where people can learn to pass the assessment honestly, and provide practice exams so people can check their learning. Rationalization is increased if people think there is no other way to pass the test than cheating.

3. Make sure that the test is trustable (reliable and valid) and fair. If the test is not seen as fair,  people will be less like to rationalize that it’s permissible to cheat.

3. Communicate details of why the tests are there, how the questions are constructed and what measures you take to make the Cheat sheet in a juice box test fair, valid and reliable. Again, if people know the test is there for good reason and fair, they will be less motivated to cheat.

4. Maintain a positive public image. This will reduce rationalization by people claiming that  the assessment provider is incompetent or has other faults.

5. Communicate your security measures and how people who cheat are caught.  This makes people less likely to think they will be able to get away with it.

For many organizations — in addition to other anti-cheating measures — it can be very productive to spend time reducing participants’ rationalization to cheat, thereby helping them choose to be honest. The picture on the right shows a “cheat sheet” or “crib sheet” hidden in a juice carton. Think of ways you can encourage participants to use their inventiveness to learn to pass the exam, not to believe it’s okay to defraud you and the testing system.

I hope you find this good practice tip helpful. I’ll be presenting at the Questionmark Users Conference March 10 – 13 on Twenty Testing Tips: Good practice in using assessments. Taking measures to reduce rationalization for cheating will be one of my tips. Register for the conference if you’re interested in hearing more.

Item Development – Planning your field test study

Austin Fossey-42Posted by Austin Fossey

Once the items have passed their final editorial review, they are ready to be delivered to participants, but they are not quite ready to be delivered as scored items. For large-scale assessments, it is best practice to deliver your new items as unscored field test items so that you can gather item statistics for review before using the items to count toward a participant’s score. We discussed field test studies in an earlier post, but today we will focus more on the operational aspects of this task.

If you are embedding field test items, there is little you need to do to plan for the field test, other than to collect data on your participants to ensure representativeness and to make sure that enough participants respond to the item to yield stable statistics. You can collect data for representativeness by using demographic questions in Questionmark’s authoring tools.

If field testing an entire form, you will need to plan your field test carefully. When an entire form is going to be field tested, Schmeiser and Welch ( Educational Measurement, 4th ed.) recommend testing twice as many items as you will need for your operational form.

To check representativeness, you may want to survey your participants in advance to help you select your participant sample. For example, if your participant population is 60% female and 40% male, but your field test sample is 70% male, then that may impact the validity of your field test results. It will be up to you to decide which factors are relevant (e.g., sex, ethnicity, age, level of education, location, level of experience). You can use Questionmark’s authoring tools and reports to deliver and analyze these survey results.

You will also need to entice participants to take your field test. Most people will not want to take a test if they do not have to, but you will likely want to conduct the field test expeditiously. You may want to offer an incentive to test, but that incentive should not bias the results.

For example, I worked on a certification assessment where the assessment cost participants several hundred dollars. To incentivize participation in the field test study of multiple new forms, we offered the assessment free of charge and told participants that their results would be scored once the final forms were assembled. We surveyed volunteers and selected a representative sample to field test each of the forms.

The number of responses you need for each item will depend on your scoring model and your organization’s policies. If using Classical Test Theory, some organizations will feel comfortable with 80 – 100 responses, but Item Response Theory models may require 200 – 500 responses to yield stable item parameters.

More is always better, but it is not always possible. For instance, if an assessment is for a very small population, you may not have very many field test participants. You will still be able to use the item statistics, but they should be interpreted cautiously in conjunction with their standard errors. In the next post, we will talk about interpreting item statistics in the psychometric review following the field test.

A sneak peek at the conference agenda

Headshot JuliePosted by Julie Delazyn

Do you want to be part of the most important learning event of the year? Come to the Questionmark Users Conference in Napa Valley March 10 – 13!

Here are some great reasons to be there:

  • Protect your assessment data: Stop cheaters, hackers and attackers
  • Examine the tools and techniques essential for trustable assessment results
  • Create better assessments: measure knowledge, skills and abilities
  • Use assessments to gather business intelligence
  • Understand the technologies that support sound assessment practices
  • Explore the latest Questionmark authoring, delivery and reporting feature

Register by January 29 for this essential professional development event and save $100.

We are thrilled by the rich content that will be offered at the conference and wanted to give you a sneak-peek at what you’ll have access to in Napa.

Here’s the current line-up of breakout sessions:

grape icon

 

Customer Stories

  • Using Questionmark to Conduct Performance Based Certifications
  • Transitioning 70 Years of High Stakes Testing to Questionmark
  • Writing Performance-Based Test Items
  • Deploying Questionmark Perception in a Heavily Firewalled Load-Balanced Environment
  • Building a Better Report with OData
  • Using Questionmark to Develop an Innovative Personnel Selection Tool
  • Questionmark AICC Integration with SuccessFactorscollage

grape icon

 

Best Practices

  • Introduction to Item Development in Large-Scale Test Development
  • Making the Case for Trustable Assessment Results in your Organization
  • Questionmark Customer Portals: Deploying Questionmark to work for your Business
  • Twenty Testing Tips: Good practice in using assessments
  • Hackers, Attackers and Your Assessments: Protecting your assessment data with penetration testing
  • Last Mile of Delivery: Ensuring the security of your assessments

grape icon

 

Questionmark Features & Functions

  • Assessment Authoring
  • Integrating with Key Technologies to Maximize Your Assessments and Results
  • Overview of Setting Performance Standards: Making the cognitive leap from scores to interpretations
  • BYOL: Item and Topic Authoring
  • Successfully Deploying Questionmark Perception
  • Customizing the Participant Interface

There will be so much to learn during our three days together. Other activities will include evening events in California’s wine country! We look forward to seeing you there.

Register for the conference by January 29th to save $100

Trusting you have a good new year

John Kleeman HeadshotPosted by John Kleeman

As we bid 2014 goodbye  and welcome 2015, we wish all readers of this blog a happy and prosperous new year.

Trust is in the news a lot these days. 2013 was memorable for its revelations of government surveillance of the Internet. Well-intentioned government organizations were intercepting Internet communications for law and order purposes, and to protect society from harm. However the surprising scale of the interceptions divided the community with some feeling it was appropriate given the threat but others becoming less trustful of government.

In 2014, we have seen a series of Internet vulnerabilities. The catchy names – Heartbleed, Shellshock and Poodle – bely the potential seriousness of these threats. Questionmark was only lightly touched by these vulnerabilities and any minor issues were quickly corrected (see Questionmark and Heartbleed and Questionmark not impacted by Bash/ShellShock Internet vulnerability). However as we’ve seen in the news, some other companies have been impacted by these or other vulnerabilities, and we are all very sensibly being more cautious about security and data protection.

Questionmark has and continues to put a high priority on security and data protection. Watch this video for more about Questionmark’s commitment to security.

2014 seems to have been the year that security and data protection have come of age. Mature organizations recognize that there are significant security threats to their data, and mature suppliers put in place extensive measures to protect against such threats. The arguments in favour of outsourcing to the Cloud remain strong; if nothing else, Cloud providers can typically protect specialist data like assessments better than a busy in-company IT team, who are focused elsewhere. But trust must be at  the forefront – you need to trust and review all your suppliers, to check that they are following good security practice. We welcome all the review we get from our customers’ IT security departments   – good questions help make us stronger.

Trust and trustable assessment results are critical to Questionmark. Our vision is that in today’s world, success for organizations, individuals and society means having the right knowledge, skills and abilities at the right place and the right time. An organization needs to know what its people understand and what they need to change or learn to meet goals. An individual needs to demonstrate achievement and find out how to improve. And society needs to know who is competent and whom to trust.

Assessments are critically needed to identify if people “know it, understand it and can do it”. Questionmark aims to provide the world’s leading online assessment service, allowing organizations to securely create, deliver and report on tests, quizzes, surveys and exams. Questionmark focuses on getting trustable results that are actionable for organizations, individuals and society.

During 2015 we’ll be sharing lots about assessment and good practice on this blog, and I trust we will have much to interest you!

 

It’s a quiz…Happy Holidays!

The holidays are here; it’s time for cheer!

No matter if you’ve been naughty or nice, take this quiz and check it twice.

Scan the QR Code or click here to get started:

 

 

Next Page »
SAP Microsoft Oracle HR-XML AAIC