Getting Results -- The Questionmark Blog » Writing Assessments

Podcast: Using Questionmark Live for Item Writing Workshops

Joan Phaup — Fri, 18 Dec 2009 13:41:28 +0000

The School of Justice at Miami Dade College in Florida plays an important role in helping people prepare for careers in public safety. Not only does the school offer degree programs, it is also home to the state’s basic abilities testing program for prospective police, correctional and probation officers.

The Florida Basic Abilities Test (F-BAT), which measures basic abilities of recruits planning to enter training programs, needs to be kept current and relevant. The College reviews the test annually and freshens the questions, drawing on the expertise of subject matter experts (SMEs) from all over the state.

The introduction this year of Questionmark Live has given the college a new, more efficient way of working with SMEs: During a recently item writing workshop held at the College, SMEs were given a quick tutorial on the use of Questionmark Live and were soon creating questions using its simple browser-based authoring tools. Their efforts were so successful that the college plans to run future workshops at a distance, doing away with the need for SMEs to meet together in order to help create test content.

You can learn more about the college’s role here and find out more about the item writing sessions by listening to the following podcast with Lebsica Gonzalez, F-BAT project manager at Miami Dade College.

Pre-Hospital Emergency Care Council Increases Certification Testing

Sarah Elkins — Wed, 09 Dec 2009 15:06:46 +0000

Posted By Sarah Elkins

We recently published a case study on how the Pre-Hospital Emergency Care Council (PHECC) are using Questionmark to do certification testing. It’s a great example of how online assessment management can make the statistical review process more efficient. By combining the easy-to-use authoring tools available in Questionmark Live and the statistical reporting features available in Questionmark Perception, PHECC have created a streamlined process that ensures valid and reliable items.

PHECC is the Irish EMS regulator, responsible for certifying all pre-hospital emergency care professionals including Emergency Medical Technicians, Paramedics and Advanced Paramedics. Having valid and reliable assessments is a crucial factor in this work. Volunteer organisations such as the Red Cross also use PHECC’s exams to certify staff on a voluntary basis. Since introducing Questionmark, PHECC have been able to dramatically increase the number of exams they have been able to provide, and they are now fully booked six months in advance.

Want to know more? Read the Case Study

Getting to Know…Tech Central

Joan Phaup — Fri, 13 Nov 2009 15:43:13 +0000

Posted by Joan Phaup

We decided at our last U.S. Users Conference in Memphis that it would be good to set aside a special place for attendees to meet with our support technicians one-on-one. Thus was born Tech Central, which was such a hit that it’s become a regular part of our conferences.

As we get ready for our 2010 Users Conference in Miami next March, I asked Ed Bell, our technical services manager, a few questions about this recent addition:

What’s Tech Central? Tech Central allows customers at our Users Conferences to meet one-on-one with the technicians. We sit down with them for 20 minutes or half an hour and work through issues we would normally talk about on the phone. Tech Central gives us a dedicated space for meeting with people. Now they know exactly where to find us instead of hoping they’ll be able to find us in passing.

How did customers respond to the first Tech Central in Memphis? You couldn’t ask for a better response. The customers could see that we really cared. The first day was sparsely attended, but once word spread we got more and more visitors. Some of them would check in with their office after visiting us and then come back again with follow-up questions. About three-fourths of the people who attended the conference came to see us. Tech Central was such a hit that we took it to the European Users Conference in England this fall.

How was Tech Central for Techs? It was hectic because we had so many people coming through, but we thought it was great! Sometimes people were queued up because it was so popular. Luckily we had a really large space with WIFI, so people could email while they waited. What made it so wonderful for us was that it gave us a chance to really connect with our customers as people. That face-to-face contact makes a big difference in our relationships with customers. When they call us on the phone, we know who they are!

What are you looking forward to most about the next Users Conference? Reconnecting with people I’ve met in the past! I enjoy seeing customers at Breakfast Briefings and our conferences. That personal bond you build from being with people is wonderful. I also look forward to seeing the customers’ reactions to the new features in Questionmark Perception, and I always like hearing customers’ success stories and seeing how enthusiastic they are. People generally call tech support when they have a problem, and it’s our job to take care of that problem, so hearing about successes does us a lot of good! And those of us who attend the conference share those success stories with our colleagues, which gives them a boost, too. I can’t wait for 2010!

If you can’t wait either, here’s a gentle reminder that earlybird-registration ends December 4th.

You can learn more about the 2010 Questionmark Users Conference at www.questionmark.com/go/conference.

Tips for preventing cheating and ensuring assessment security

Julie Chazyn — Mon, 02 Nov 2009 18:46:09 +0000

Posted by Julie Chazyn

Last week I wrote about tips for protecting intellectual property. It’s equally important keep people from cheating on tests, so here are three pointers on that subject. I’ll be following this up with more tips in future posts. Leave me your comments; we can always add to lists like these!

.Screening tests

Consider givinig a small pre-screening test to prevent people from taking an assessment that is beyond their current ability level. If a participant can‘t answer a certain number of these questions correctly they will not be allowed to see the remainder of the assessment. When the time does come for them to take the test, they will not have already seen its content.

Candidate agreements

Candidate agreements or examination honor codes require a participant’s agreement before they start an assessment, say by clicking on an “OK “ or “Yes” button after reading the exam’s code of conduct.

The code might say something like this: “I agree to answer the questions on this assessment without obtaining assistance from another person or via electronic means. I agree to not to share my answers with anyone during or after the exam. I further agree to not memorize or otherwise steal the intellectual property contained in this exam. I accept that if any of these conditions are violated, my exam results will be set to a zero, I will not be able to retake the exam for a period of 10 years, and I may be charged with a crime under regional laws.”

Here are some topics you might want to cover in a candidate agreement:

The test vendor will have the option to terminate the assessment if suspicious behavior is detected
The candidate must abide by the rules of the test center, organization, or program
The candidate will not provide false ID or false papers
The candidate cannot take the test on behalf of someone else
The candidate will not engage in cheating in any form
The candidate will not help others cheat by disclosing information about the assessment
The candidate will not use aids that are not allowed
The candidate will not solicit someone else to take the test
The candidate will not cause a disturbance in the testing center
The candidate will not tamper with the test center in any way
The candidate will not share information

Limiting content exposure/leakage

In order to limit the amount of question content being shown to a participant at any given time, think about using question-by-question templates. These present questions one at a time to participants so that exam content is not completely exposed on screen. Participants who may intend to take pictures of the exam content or otherwise steal intellectual property will not be able to do so all at once.

There are many fine resources for learning how to prevent cheating. Two of thes are books by Dr. Gregory Cizek: Cheating on Tests: How to Do It, Detect It and Prevent It and Detecting and Preventing Classroom Cheating: Promoting Integrity in Assessment.

There’s also our white paper: “Delivering Assessments Safely and Securely,” and of course this blog! Watch for more security tips in my future posts.

4 Tips to Help Ensure the Security of Intellectual Property

Julie Chazyn — Fri, 30 Oct 2009 14:25:47 +0000

Posted by Julie Chazyn

Protecting the intellectual property contained in a test or exam is essential, not only because of the time, effort and cost of creating assessments but also because IP theft undermines the accurate measurement of knowledge and skills.

Protecting intellectual property protects the credibility of tests. Here are four tips for helping to ensure the security of intellectual property:

Create and administer multiple test forms

Rather than having only one form of the assessment being administered, delivering multiple forms of the same exam can help limit item exposure. This method also allows for the possibility of interspersing large-scale integrated beta test questions within the forms to collect psychometric information on newly developed questions.

Restrict and control administration of beta test items

Beta testing questions is an important part of high-stakes assessment, ensuring the psychometric quality of questions before they appear on actual assessments. However, it is vital that a well conceptualized beta test model is in effect to limit the exposure of newly developed questions to participants.

Update exam forms periodically

Letting exam forms become stale can over-expose questions to participants, increasing the likelihood of IP theft. An organization could consider retiring old exam forms and turning them into exam prep materials that can be sold to participants. In this way, participants could periodically expect new practice questions.

Produce exam prep materials

Organizations should consider making exam prep materials available to participants before an assessment. This will help reduce the demand for participants to try to obtain exam questions via illegal means as they will have access to the type of questions that will be asked on the actual assessment.

For more details on this subject, plust information about various means for deploying a wide range of assessment types with assurance, download our White Paper: Delivering Assessments Safely and Securely.

5 Steps for Designing Appropriate Learning Experiences

Julie Chazyn — Mon, 26 Oct 2009 15:03:32 +0000

Posted by Julie Chazyn

Assessments provide a valuable tool for helping organizations properly design effective and useful learning experiences. Doing so involves a five-step process.

Step 1: Define the objectives. An objective might be to increase customer satisfaction, reduce error rates or improve safety.

Step 2: Ask what knowledge and skills are required to meet the objectives. In a college or university course on organic chemistry, for example, it’s important to ask what are the knowledge and skills required to comprehend those concepts and use them. The answers to that question will help the professor establish the topic structure to define the knowledge, skills, abilities and attitudes required to meet the objectives.

Step 3: Run a needs analysis survey or skills gap survey. Here, people take a needs assessment to reveal the knowledge and skills they already have as well as what they still need. A gap analysis can be derived from the difference between what is required and people’s current knowledge and skills.

Step 4: Develop a learning plan. That plan will describe the learning objectives and explain how the plan will be administered. The learning objectives will guide the production of learning materials and assessments. Facilitating the learning process might involve instructor-led training, coaching by managers, or e-learning courses.

Step 5: Conduct a pre-learning assessment. The pre-learning assessment will have two purposes: to create intrigue about the course and to guide each participant to the right learning experience. Advanced and novice learners will require different approaches.

To learn more about how instructors and organizations can use assessments to improve learning, download the white paper Assessments through the Learning Process.

Item Analysis Analytics: The White Paper

Greg Pope — Fri, 23 Oct 2009 14:10:51 +0000

Posted by Greg Pope

I had a great time putting together an eight-part series on Item Analysis Analytics for this blog and was pleased with the interest it received.

When a reader asked if it would be possible to present all the posts in a single document I thought hey, let’s present the content of these articles in the form of a Questionmark White Paper! So here it is for you to download with our compliments.

I hope the paper helps you in your efforts to create test questions that make the grade!

Measuring Learning Results: Eight Recommendations for Assessment Designers

Joan Phaup — Mon, 19 Oct 2009 14:28:37 +0000

Posted by Joan Phaup

Is it possible to build the perfect assessment design? Not likely, given the intricacies of the learning process! But a white paper available on the Questionmark Web site helps test authors respond effectively to the inevitable tradeoffs in order to create better assessments.

Measuring Learning Results, by Dr. Will Thalheimer of Work-Learning Research, considers findings from fundamental learning research and how they relate to assessment. The paper explores how to create assessments that measure how well learning interventions are preparing learners to retrieve information in future situations—which as Will states it is the ultimate goal of training and education.

The eight bits of wisdom that conclude the paper give plenty of food for thought for test designers. You can download the paper to find out how Will arrived at them.

1. Figure out what learning outcomes you really care about. Measure them. Prioritize the importance of the learning outcomes you are targeting. Use more of your assessment time on high-priority information.

2. Figure out what retrieval situations you are preparing your learners for. Create assessment items that mirror or simulate those retrieval situations.

3. Consider using delayed assessments a week or month (or more) after the original learning ends—in addition to end-of-learning assessments.

4. Consider using delayed assessments instead of end-of-learning assessments, but be aware that there are significant tradeoffs in using this approach.

5. Utilize authentic questions, decisions, or demonstrations of skill that require learners to retrieve information from memory in a way that is similar to how they’ll have to retrieve it in the retrieval situations for which you are preparing them. Simulation-like questions that provide realistic decisions set in real-world contexts are ideal.

6. Cover a significant portion of the most important learning points you want your learners to understand or be able to utilize. This will require you to create a list of the objectives that will be targeted by the instruction.

7. Avoid factors that will bias your assessments. Or, if you can’t avoid them, make sure you understand them, mitigate them as much as possible, and report their influence. Beware of the biasing effects of end-of-learning assessments, pretests, assessments given in the learning context, and assessment items that are focused on low-level information.

8. Follow all the general rules about how to create assessment items. For example, write clearly, use only plausible alternatives (for multiple-choice questions), pilot-test your assessment items to improve them, and utilize psychometric techniques where applicable.

Bringing Precision to Industrial Certifications

Joan Phaup — Fri, 16 Oct 2009 14:36:31 +0000

Posted by Joan Phaup

Recently I learned how skilled employees at one manufacturing company demonstrate their understanding of blueprints and measurements as part of their certification program. These workers need to be precise—down to tolerances as narrow as a human hair!

How to measure employee’s accuracy and understanding? Scan blueprints and quality documents from job orders into Questionmark Perception and then pose questions about them.

Some questions require participants to identify the exact place on the blueprint where particular information should be found. Others ask them to identify different elements within the blueprint or enter a calculated value such as a maximum limit interpreted from the print.

It used to be that engineering staff would conduct time-consuming, one-on-one certifications with workers. In addition to saving time, the online tests offer greater objectivity than the one-on-one exams. They also provide a “blueprint” of sorts for future learning: If someone doesn’t pass a certification, a supervisor reviews the Perception Coaching Report from the exam to identify problem areas. Excel files recording employees’ performance on all of the questions help identify weaknesses in their collective understanding, prompting trainers to focus on those areas.

Want to know more? Read our case study.