Content protection and secure delivery: Test Design and Delivery Part 9

Posted By Doug Peterson

Writing good items and putting together valid and reliable assessments can take a lot of time and cost a lot of money. Part of an assessment’s reliability and validity is based on the test-taker not knowing the items ahead of time. For these reasons, it is critical that item exposure be controlled.

This starts during the development process by requiring everyone who develops items or assessments to sign a confidentiality agreement. Developers’ computers should, at the very least, be password-protected, and you should consider data encryption as well.

Care must be taken to prevent question theft once an assessment is assembled and delivered. Do not allow overly generous time limits, which would provide time for a test-taker to go back through the assessment and memorize questions. If your assessment is delivered electronically, consider not allowing backward movement through the test. Be very careful about allowing the use of a “scribble sheet”, as someone might try to write down questions and sneak them out of the test center: be sure to number all scribble sheets and collect them at the end of the assessment.

Computer-based testing makes it very easy to utilize random item selection when the assessment is delivered. While this does mean having to develop more items, it cuts down the number of times any one item is delivered and helps to reduce cheating by presenting different questions in a different order to teach test-taker.

It is critical to track the number of times an item has been delivered. After a certain number of deliveries, you will want to retire an item and replace it with a new item. The main factor that impacts how many times an item should be exposed is whether the assessment is high-stakes or low-stakes. Items on a high-stakes exam should have a lower maximum number of exposures, but items on a low-stakes exam can have a higher number of exposures.

As long as there have been tests, there have been test-takers who try to cheat. Make sure that you authenticate each examinee to ensure that the person who is supposed to be taking the exam is, in fact, the person taking the exam. Testing centers typically prohibit talking, using notes, and using cell phones during tests. Maintain a minimum amount of space between test-takers, or use carrels to physically separate them.

Test administrators should walk around the room during the test. Unauthorized personnel should not be permitted to enter the room during the test, and the administrator should not leave the room for any reason without first bringing in another administrator.

Computer-based testing presents its own set of security challenges, especially when testing is permitted outside of a secure testing center (e.g., in the test-taker’s home). Questionmark offers the Questionmark Secure client, which locks  down test-takers’ machines and doesn’t allow them to copy questions or switch tasks.

Computer-based testing security can/should also include some form of identification and password verification. Additionally, in the last few years, technology has become available that allows for the remote monitoring of test-takers using built-in laptop/tablet cameras or small desktop devices.

Click here for links to a complete listing of posts in this series.

Final Planning Considerations – Test Design and Delivery Part 3

Posted By Doug Peterson

In Part 2 of this series, we looked at how to determine how many items you needed to write for each content area covered by your assessment. In this installment, we’ll take a look at a few more things that must be considered before you start writing items.

You must balance the number of items you’ve determined that you need with any time constraints imposed on the actual taking of the test. Take a look at the following table showing the average amount of time a participant spends on different question types:

It’s easy to see that it will take much longer for a participant to take an assessment containing 100 short-answer questions than one with 100 True/False questions. Therefore a time limit on the assessment might constrain what question types can be used, or conversely, the time limit may be influenced by how many questions you calculate you need and the question types you want to use. There’s no hard and fast rule here, it’s just something that needs to be considered before you go off to write a bunch of items.

The time a participant requires to complete an item is not the only thing that should be considered when determining the item format for your assessment. True/False questions might have some problems because there is a 50% chance of guessing correctly, making it hard to know if the participant knew the material or was just lucky. (I look at this in more depth in this blog article.)

Multiple Choice questions are good for recall and are easy, quick and objective to score, but many assessment developers feel that they really only test the participant’s ability to memorize. Some learning professionals are critical about incorrect information being presented, which could lead to the wrong answer being recalled at some point in the future. I discuss writing Multiple Choice question here  and  here.

Essay questions are good for testing higher cognitive skills such as formulating a correct answer from scratch and applying concepts to a new situation. However, they take longer for the participant to answer, and scoring takes longer and is more subjective than for other question types.

Finally, you need to decide on the presentation format, which boils down to basically two choices: paper and pencil, or computer-based (including over the Internet). Each has their pros and cons.

Paper and pencil is not susceptible to technical problems. It is comfortable for people unfamiliar with computers. However, it’s labor-intensive when it comes to distribution of the materials, grading, tracking, reporting, etc.

Computer-based assessments are faster and easier to update. Results can be provided immediately. Computer-based assessments allow for question types such as software simulations that paper and pencil can’t provide. However, not everyone is comfortable with computers or can type at a reasonable rate – you don’t want someone who knows the material to fail a test because they couldn’t answer questions fast enough using a keyboard.

Embedding Questionmark Assessments in Google Wave

Screenshot of an embedded assessment in Google WaveEmbed a Questionmark Perception assessment, survey or quiz inside your Google Wave profile.

  • To see how this would look, see a snapshot of an assessment embedded into Google Wave.
  • Check out this How-to on our developer Web site.
  • Google Wave is an online tool for real-time communication and collaboration.  Embedding an assessment into Google Wave may be useful if you want to ask the members of your Wave to complete a quiz or  simply fill in a survey. The results can then be analyzed and reported on from Perception.

Podcast: Global Skills-Based Certification

Posted by Joan Phaup

I chatted with Adam Zaller from Services University of NCR during the Questionmark 2009 Users Conference. Adam told me about the university’s global skills-based certification program for customer engineers who repair equipment such as ATMs in more than 45 countries. From day one, the most experienced engineers have helped design and build the program. That includes helping to provide test content.

Listen in on our conversation to find out more.

Do You Know How to Write Good Test Questions?


Posted by Howard Eisenberg

I had a typical education.  I took lots of tests.  Knowing what I know now about good testing practice, I wonder how many of those tests really provided an accurate measure of my knowledge.

Common testing practices often contradict what is considered best practice.  This piece will focus on four of the most common “myths” or “mistakes” that teachers, subject matter experts, trainers and educators in general make when writing test questions.

1) A multiple choice question must have at least four choices.  False.
Three to five choices is considered sufficient.  Of course the fewer the choices, the greater the chance a test-taker can guess the correct answer.  However, the point however is you don’t need four choices, and if you are faced with the decision of adding an implausible or nonsensical distracter to make four choices, it won’t add any measurement value to the question anyway.  Might as well just leave it at three choices.

2)  The use of “all of the above” as a choice in a multiple choice question is good practice.  False.
It may be widely used but it is poor practice.  “All of the above” is almost always the correct answer.  Why else would it be there?  It is tacked onto a multiple choice question so it can have only one best answer. After all, writing plausible distracters is difficult.  If at least two of the other choices answer the question, then “all of the above” is the answer. No need to consider any more choices.

3) Starting a question with “Which of the following is not …” is considered best practice.  False.

First, the use of negatives in test questions should be avoided (unless you are trying to measure a person’s verbal reasoning ability).  Second, the use of the “which of the following …” form usually results in a question that only tests basic knowledge or recall of information presented in the text or in the lecture.  You might as well be saying:  “Which of the following sentences does not appear exactly as it did in the manual?

A) Copy > paste (from manual) choice 1
B) Copy > past choice 2
C) Copy > past choice 3
D) Make something up

While that may have some measurement value, my experience tells me that most test writers prefer to measure how well a person can apply knowledge to solve novel problems.  This type of question just won’t reach that level of cognition.  If you really want to get to problem-solving, consider using a real-world scenario and then posing a question.

4) To a subject matter expert, the correct answer to a good test question should be apparent.  True.

A subject matter expert knows the content.  A person who really knows the content should be able to identify the best answer almost immediately.  Test writers often hold the misconception that a good test question is one that is tricky and confusing.  No, that’s not the point of a test.  The point is to attain an accurate measure of how well a person knows the subject matter or has mastered the domain.  The question should not be written to trick the test-taker, let alone the expert. That just decreases the value of the measurement.

There are many more “do’s” and “don’ts” when it comes to writing good test questions.  But you can start to improve your test questions now by considering these common misconceptions as you write your next test.

12 Tips for Writing Good Test Questions

Posted by Joan Phaup

Writing effective questions takes time and practice. Whether your goal is to measure knowledge and skills, survey opinions and attitudes or enhance a learning experience, poorly worded questions can adversely affect the quality of the results.

I’ve gleaned the following tips for writing and reviewing questions from Questionmark’s learning resources:

1. Keep stems and statements as short as possible and use clear, concise language.toolbox
2. Use questions whenever possible (What, Who, When, Where, Why and How).
3. Maintain grammatical consistency to avoid cueing.
4. List choices in a logical order.
5. Avoid negatives, especially double negatives.
6. Avoid unnecessary modifiers, especially absolutes (e.g. always, never, etc.).
7. Avoid “All of the above” and use of “None of the above” with caution.
8. Avoid vague pronouns (e.g. it, they).
9. Avoid conflicting alternatives.
10. Avoid syllogistic reasoning choices (e.g. “both a and b are correct”) unless absolutely necessary.
11. Avoid providing cues to correct answer in the stem.
12. Avoid providing clues to the answer of one question in another question.

If you would like more information about writing question and assessments, a good place to start is the Questionmark Learning Cafe.