We decided at our last U.S. Users Conference in Memphis that it would be good to set aside a special place for attendees to meet with our support technicians one-on-one. Thus was born Tech Central, which was such a hit that it’s become a regular part of our conferences.

As we get ready for our 2010 Users Conference in Miami next March, I asked Ed Bell, our technical services manager, a few questions about this recent addition:

What’s Tech Central? Tech Central allows customers at our Users Conferences to meet one-on-one with the technicians. We sit down with them for 20 minutes or half an hour and work through issues we would normally talk about on the phone. Tech Central gives us a dedicated space for meeting with people. Now they know exactly where to find us instead of hoping they’ll be able to find us in passing.

How did customers respond to the first Tech Central in Memphis? You couldn’t ask for a better response. The customers could see that we really cared. The first day was sparsely attended, but once word spread we got more and more visitors. Some of them would check in with their office after visiting us and then come back again with follow-up questions. About three-fourths of the people who attended the conference came to see us. Tech Central was such a hit that we took it to the European Users Conference in England this fall.

How was Tech Central for Techs? It was hectic because we had so many people coming through, but we thought it was great! Sometimes people were queued up because it was so popular. Luckily we had a really large space with WIFI, so people could email while they waited. What made it so wonderful for us was that it gave us a chance to really connect with our customers as people.  That face-to-face contact makes a big difference in our relationships with customers. When they call us on the phone, we know who they are!

What are you looking forward to most about the next Users Conference? Reconnecting with people I’ve met in the past! I enjoy seeing customers at Breakfast Briefings and our conferences. That personal bond you build from being with people is wonderful. I also look forward to seeing the customers’ reactions to the new features in Questionmark Perception,  and I always like hearing customers’ success stories and seeing how enthusiastic they are. People generally call tech support when they have a problem, and it’s our job to take care of that problem, so hearing about successes does us a lot of good! And those of us who attend the conference share those success stories with our colleagues, which gives them a boost, too.  I can’t wait for 2010!

If you can’t wait either, here’s a gentle reminder that earlybird-registration ends December 4th.

You can learn more about the 2010 Questionmark Users Conference at www.questionmark.com/go/conference.

Last week I wrote about tips for protecting  intellectual property.  It’s equally important keep people from cheating on tests, so here are three pointers on that subject.  I’ll be following this up with more tips in future posts.  Leave me your comments; we can always add to lists like these!

.Screening tests

Consider givinig a small pre-screening test to prevent people from taking an assessment that is beyond their current ability level. If a participant can‘t answer a certain number of these questions correctly they will not be allowed to see the remainder of the assessment. When the time does come for them to take the test,  they will not have already seen its content.

Candidate agreements

Candidate agreements or examination honor codes require  a participant’s agreement before they start an assessment, say by clicking on an “OK “ or  “Yes” button after reading the exam’s code of conduct.

The code might say something like this:  “I agree to answer the questions on this assessment without obtaining assistance from another person or via electronic means. I agree to not to share my answers with anyone during or after the exam. I further agree to not memorize or otherwise steal the intellectual property contained in this exam. I accept that if any of these conditions are violated, my exam results will be set to a zero, I will not be able to retake the exam for a period of 10 years, and I may be charged with a crime under regional laws.”

Here are some topics you might want to cover in a candidate agreement:

• The test vendor will have the option to terminate the assessment if suspicious behavior is detected
• The candidate must abide by the rules of the test center, organization, or program
• The candidate will not provide false ID or false papers
• The candidate cannot take the test on behalf of someone else
• The candidate will not engage in cheating in any form
• The candidate will not help others cheat by disclosing information about the assessment
• The candidate will not use aids that are not allowed
• The candidate will not solicit someone else to take the test
• The candidate will not cause a disturbance in the testing center
• The candidate will not tamper with the test center in any way
• The candidate will not share information

Limiting content exposure/leakage

In order to limit the amount of question content being shown to a participant at any given time, think about using question-by-question templates. These present questions one at a time to participants so that exam content is not completely exposed on screen. Participants who may intend to take pictures of the exam content or otherwise steal intellectual property will not be able to do so all at once.

There are many fine resources for learning how to prevent cheating. Two of thes are books by Dr. Gregory Cizek:  Cheating on Tests: How to Do It, Detect It and Prevent It and Detecting and Preventing Classroom Cheating: Promoting Integrity in Assessment.

There’s also our white paper: “Delivering  Assessments Safely and Securely,” and of course this blog! Watch for more security tips in my future posts.

Protecting the intellectual property contained in a test or exam is essential, not only because of the time, effort and cost of creating assessments but also because IP theft undermines the accurate measurement of knowledge and skills.

Protecting intellectual property protects the credibility of tests. Here are four tips for helping to ensure the security of intellectual property:

Create and administer multiple test forms

Rather than having only one form of the assessment being administered, delivering multiple forms of the same exam can help limit item exposure. This method also allows for the possibility of interspersing large-scale integrated beta test questions within the forms to collect psychometric information on newly developed questions.

Restrict and control administration of beta test items

Beta testing questions is an important part of high-stakes assessment, ensuring the psychometric quality of questions before they appear on actual assessments. However, it is vital that a well conceptualized beta test model is in effect to limit the exposure of newly developed questions to participants.

Update exam forms periodically

Letting exam forms become stale can over-expose questions to participants, increasing the likelihood of IP theft. An organization could consider retiring old exam forms and turning them into exam prep materials that can be sold to participants. In this way, participants could periodically expect new practice questions.

Produce exam prep materials

Organizations should consider making exam prep materials available to participants before an assessment. This will help reduce the demand for participants to try to obtain exam questions via illegal means as they will have access to the type of questions that will be asked on the actual assessment.

For more details on this subject, plust information about various means for deploying a wide range of assessment types with assurance, download our White Paper: Delivering Assessments Safely and Securely.

Assessments provide a valuable tool for helping organizations properly design effective and useful learning experiences. Doing so involves a five-step process.

Step 1: Define the objectives. An objective might be to increase customer satisfaction, reduce error rates or improve safety.

Step 2: Ask what knowledge and skills are required to meet the objectives. In a college or university course on organic chemistry, for example, it’s important to ask what are the knowledge and skills required to comprehend those concepts and use them. The answers to that question will help the professor establish the topic structure to define the knowledge, skills, abilities and attitudes required to meet the objectives.

Step 3: Run a needs analysis survey or skills gap survey. Here, people take a needs assessment to reveal the knowledge and skills they already have as well as what they still need. A gap analysis can be derived from the difference between what is required and people’s current knowledge and skills.

Step 4: Develop a learning plan. That plan will describe the learning objectives and explain how the plan will be administered. The learning objectives will guide the production of learning materials and assessments. Facilitating the learning process might involve instructor-led training, coaching by managers, or e-learning courses.

Step 5: Conduct a pre-learning assessment. The pre-learning assessment will have two purposes: to create intrigue about the course and  to guide each participant to the right learning experience. Advanced and novice learners will require different approaches.

To learn more about  how instructors and organizations can use assessments to improve learning, download the white paper Assessments through the Learning Process.

Posted by Greg Pope

I had a great time putting together an eight-part series on Item Analysis Analytics for this blog and was pleased with the interest it received.

When a reader asked if it would be possible to present all the posts in a single document I thought hey, let’s present the content of these articles in the form of a Questionmark White Paper! So here it is for you to download with our compliments.

I hope the paper helps you in your efforts to create test questions that make the grade!

Is it possible to build the perfect assessment design? Not likely, given the intricacies of the learning process! But a white paper available on the Questionmark Web site helps test authors respond effectively to the inevitable tradeoffs in order to create better assessments.

Measuring Learning Results, by Dr. Will Thalheimer of Work-Learning Research, considers findings from fundamental learning research and how they relate to assessment. The paper explores how to create assessments that measure how well learning interventions are preparing learners to retrieve information in future situations—which as Will states it is the ultimate goal of training and education.

The eight bits of wisdom that conclude the paper give plenty of food for thought for test designers. You can download the paper to find out how Will arrived at them.

1. Figure out what learning outcomes you really care about. Measure them. Prioritize the importance of the learning outcomes you are targeting. Use more of your assessment time on high-priority information.

2. Figure out what retrieval situations you are preparing your learners for. Create assessment items that mirror or simulate those retrieval situations.

3. Consider using delayed assessments a week or month (or more) after the original learning ends—in addition to end-of-learning assessments.

4. Consider using delayed assessments instead of end-of-learning assessments, but be aware that there are significant tradeoffs in using this approach.

5. Utilize authentic questions, decisions, or demonstrations of skill that require learners to retrieve information from memory in a way that is similar to how they’ll have to retrieve it in the retrieval situations for which you are preparing them. Simulation-like questions that provide realistic decisions set in real-world contexts are ideal.

6. Cover a significant portion of the most important learning points you want your learners to understand or be able to utilize. This will require you to create a list of the objectives that will be targeted by the instruction.

7. Avoid factors that will bias your assessments. Or, if you can’t avoid them, make sure you understand them, mitigate them as much as possible, and report their influence. Beware of the biasing effects of end-of-learning assessments, pretests, assessments given in the learning context, and assessment items that are focused on low-level information.

8. Follow all the general rules about how to create assessment items. For example, write clearly, use only plausible alternatives (for multiple-choice questions), pilot-test your assessment items to improve them, and utilize psychometric techniques where applicable.

Recently I learned how skilled employees at one manufacturing company demonstrate their understanding of blueprints and measurements as part of their certification program. These workers need to be  precise—down to tolerances as narrow as a human hair!

How to measure employee’s accuracy and understanding? Scan blueprints and quality documents from job orders into Questionmark Perception and then pose questions about them.

Some questions require participants to identify the exact place on the blueprint where particular information should be found. Others ask them to identify different elements within the blueprint or enter a calculated value such as a maximum limit interpreted from the print.

It used to be that engineering staff  would conduct time-consuming, one-on-one certifications with workers. In addition to saving time, the online tests offer greater objectivity than the one-on-one exams. They also provide a “blueprint” of sorts for future learning: If someone doesn’t pass a certification, a supervisor reviews the Perception Coaching Report from the exam to identify problem areas.  Excel files recording employees’ performance on all of the questions help identify weaknesses in their collective understanding, prompting trainers to focus on those areas.

Want to know more? Read  our case study.

Today I’d like to share this chart that can help you determine what delivery environment would be suitable for various types of assessments. The chart appears in our white paper, Delivering Assessments Safely and Securely, which offers many more details on this subject–the aim being to help you avoid incurring unnecessary costs while at the same time giving your assessments the security levels they require.

Posted by Greg Pope

In my last post, I showed a few more examples of item analyses where we drilled down into why some questions had problems. I thought it might be useful  to show a few examples of some questions that have bad and downright terrible psychometric performance to show the ugly side of item analysis.

Below is an example of a question that is fairly terrible in terms psychometric performance. Here are some reasons why:

• Going from left to right, first we see that the “Number of Results” is 65, which is not so good: there are too few participants in the sample to be able to make sound judgements about the psychometric performance of the question

• Next we see that 25 participants didn’t answer the question (“Number not Answered” = 25), which means there was a problem with people not finishing or finding the questions confusing and giving up.

• The “P Value Proportion Correct” shows us that this question is hard with 20% of participants ‘getting it right.’

• The “Item Discrimination” indicates very low discrimination, with the difference between the Upper and Lower group in terms of the proportion selecting the correct answer of ‘More than 40’ at only 5%. This means that of the participants with high overall exam scores, 27% selected the correct answer versus 22% of the participants with the lowest overall exam scores. This is a very small difference between the Upper and Lower groups. Participants who know the material should have got the question right more often.

• The “Item Total Correlation” reflects the Item Discrimination with a negative value of -0.01. A value like this would definitely not meet most organizations’ internal criteria in terms of what is considered an acceptable item. Negative item-total correlations are a major red flag!

• Finally we look at the Outcome information to see how the distracters perform. We find that participants are all over the map selecting distracters in an erratic way. When I look at the question wording I realize how vague and arbitrary this question is: the number of questions that should be in an assessment depends on numerous factors and contexts. It is impossible to say that in any context a certain number of questions are required. It looks like the Upper Group are selecting the response options ‘21-40’ and ‘More than 40’ response options more than the other two options, which have smaller numbers of questions. This makes sense from a participant guessing perspective, because in many assessment contexts having more questions than fewer questions is better for reliability.

The psychometricians, SMEs, and test developers reviewing this question would need to send the SME who wrote this question back to basic authoring training to ensure that they know how to write questions that are clear and concise. This question does not really have a correct answer and needs to be re-written to clarify the context and provide many more details to the participants. I would even be tempted to throw out questions along this content line, because how long an assessment should be has no one “right answer.” How long an assessment should be depends on so many things that there will always be room for ambiguity, so it would be quite challenging to write a question that performs well statistically on this topic.

Below is an example of a question that is downright awful in terms psychometric performance. Here are some reasons why:

• Going from left to right, first we see that the “Number of Results” is 268, which is really good. That is a nice healthy sample. Nothing wrong here, let’s move on.

• Next we see that 56 participants didn’t answer the question (“Number not Answered” = 56), which means there was a problem with people not finishing or finding the questions confusing and giving up. It gets worse, much, much worse.

• The “P Value Proportion Correct” shows us that this question is really hard, with 16% of participants ‘getting it right.’

• The “Item Discrimination” indicates a negative discrimination, with the difference between the Upper and Lower group in terms of the proportion selecting the correct answer of ‘44123’ at  -23%. This means that of the participants with high overall exam scores, 12% selected the correct answer versus 35% of the participants with the lowest overall exam scores. What the heck is going on? This means that participants with the highest overall assessment scores are selecting the correct answer LESS OFTEN than participants with the lowest overall assessment scores. That is not good at all; lets dig deeper.

• The “Item Total Correlation” reflects the Item Discrimination with a large negative value of -0.26. This is a clear indication that there is something incredibly wrong with this question.

• Finally we look at the Outcome information to see how the distracters perform. This is where the true psychometric horror of this question is manifested. There is neither rhyme nor reason here: participants, regardless of their performance on the overall assessment, are all over the place in terms of selecting response options. You might as well have blindfolded everyone taking this question and had them randomly select their answers. This must have been extremely frustrating for the participants who had to take this question and would have likely led to many participants thinking that the organization administering this question did not know what they were doing.

The psychometricians, SMEs, and test developers reviewing this question would need to provide a pink slip to the SME who wrote this question immediately. Clearly the SME failed basic question authoring training. This question makes no sense and was written in such a way to suggest that the author was under the influence, or otherwise not in a right state of mind, when crafting this question. What is this question testing? How can anyone possibly make sense of this and come up with a correct answer? Is there a correct answer? This question is not salvageable and should be stricken from the Perception repository without a second thought. A question like this should have never gotten in front of a participant to take, let alone 268 participants. The panel reviewing questions should review their processes to ensure that in the future questions like this are weeded out before an assessment goes out live for people to take.