Two great workshops: Questionmark Boot Camp and Criterion-Referenced Test Development

Rick Ault

Posted by Joan Phaup

Planning for the Questionmark 2013 Users Conference in Baltimore, Maryland, March 3 – 6 is well underway.

We have begun posting descriptions of breakout sessions and are pleased to announce the return — by popular demand — of two pre-conference workshops.

Both of these all-day sessions will take place on Sunday March 3, 2013:

Questionmark Boot Camp: Basic Training for Beginners, with Questionmark Trainer Rick Ault

Learn the basics of using Questionmark technologies before the conference starts. Beginning Questionmark users are invited to bring their laptops to a basic training course.

Get into gear with hands-on practice in creating questions, putting together an assessment, then scheduling it, taking it and seeing the results. Start off with some first-hand experience that will give you a firm footing for learning more at the conference.

sharon shrock

Criterion-Referenced Test Development, with Sharon Shrock and Bill Coscarelli

Bill Coscerelli

Sharon and Bill are the authors of Criterion-Referenced Test Development: Technical and Legal Guidelines for Corporate Training. Participants in their pre-conference workshop, which is based on the book, will explore testing best practices and will learn how to meet rigorous competency testing standards and interpret test results correctly.

This workshop is ideal for trainers, instructional designers, course developers and training managers. Understanding the principles of skillful test authoring will help you create ethical and dependable testing programs that yield meaningful, measurable results.

You can save $200 by registering for the conference on or before November 16th. You can sign up for a workshop at the same time or add in a workshop later. It’s up to you!

Actionable Data and the A-model: Video

Posted By Doug Peterson

The A-model is a powerful tool for defining and solving business problems. To properly measure and evaluate a solution, you need trusted and valid information. In this video, we’ll explore how Questionmark assessments and reports provide exactly that: trusted and valid information in the measurement and evaluation phase of this practical framework.

For more details, download this white paper: Alignment, Impact and Measurement with the A-model.

Assessment types and their uses: summative assessments

Posted by Julie Delazyn

To use assessments effectively, it’s important to understand their context and uses within the learning process.

Over the past few weeks I have written about diagnostic assessments, formative assessments and needs assessments. My last post in this series is about summative assessments.

Typical uses:

  • Measuring or certifying knowledge, skills and aptitudes (KSAs)
  • Providing a quantitative grade and making a judgment about a person’s knowledge, skills and achievement
  • Determining whether the examinee meets the predetermined standard for specialized expertise
  • Determining a participant’s level of performance at a particular time


  • Licensing exams
  • Certification tests
  • Pre-employment tests
  • Academic entrance exams
  • Post-course tests
  • Exams during study

Medium, High


Summative assessments are easy to explain: they sum up the knowledge or the skills of the person taking the test. This type of assessment provides a quantitative grade and makes a judgment about a person’s knowledge, skills and achievement. A typical example would be a certification that a technician must pass in order to install and/or do repairs on a particular piece of machinery. In passing the certification exam, a candidate proves his or her understanding of the machinery.

For more details about assessments and their uses check out the white paper, Assessments Through the Learning Process. You can download it free here, after login. Another good source for testing and assessment terms is our glossary.

Assembling the Test Form — Test Design and Delivery Part 7

Posted By Doug Peterson

In the previous post in this series, we looked at putting together assessment instructions for both the participant and the instructor/administrator. Now it’s time to start selecting the actual questions.

Back in Part 2 we discussed determining how many items needed to be written for each content area covered by the assessment. We looked at writing 3 times as many items as were actually needed, knowing that some would not
make it through the review process. Doing this also enables you to create multiple forms of the test, where each form covers the same concepts with equivalent – but different – questions. We also discussed the amount of time a participant needs to answer each question type, as shown in this table:

As you’re putting your assessment together, you have to account for the time required to take the assessment. You have to multiply the number of each question type in the assessment by the values in the table above.

You also need to allow time for:

  • Reading the instructions
  • Reviewing sample items
  • Completing practice items
  • Completing demographic info
  • Taking breaks

If you already know the time allowed for your assessment, you may have to work backwards or make some compromises. For example, if you know that you only have one hour for the assessment, and you have a large amount of content to cover, you may want to consider focusing on multiple choice and fill-in-the-blank questions, and stay away from matching and short-answer to maximize the number of questions you can include in the time period allowed.

To select the actual items for the assessment, you may want to consider using a Test Assembly Form, which might look something like this:

The content area is in the first column. The second column shows how many questions are needed for that content area (as calculated back in Part 2). Each item should have a short identifier associated with it, and this is provided in the “Item #” column. The “Keyword” column is just that – one or two words to remind you what the question addresses. The last column lists the item number of an alternate item in case a problem is found with the first selection during assessment review.

As you select items, watch out for two things:

1. Enemy items. This is when one item gives away the answer to another item. Make sure that the stimulus or answer to one item does not answer or give a clue to the answer of another item.

2. Overlap. This is when two questions basically test the same thing. You want to cover all of the content in a given content area, so each question for that content area should cover something unique. If you find that you have several questions assessing the same thing, you may need to write some new questions or you may need to re-calculate how many questions you actually need.

Once you have your assessment put together, you need to calculate the cutscore. This topic could easily be another (very lengthy) blog series, and there are many books available on calculating cutscores. I recently read the book, Cutscores: A Manual for Setting Standards of Performance on Educational and Occupational Tests, by Zieky, Perie and Livingston. I found it to be a very good book, considering that the subject matter isn’t exactly “thrill a minute”. The authors discuss 18 different methods for setting cutscores, including which methods to use in various situations and how to carry out a cutscore study. They look at setting cutscores for criterion-referenced assessments (where performance is judged against a set standard) as well as norm-referenced assessments (where the performance of one participant is judged against the performance of the other participants). They also look at pass/fail situations as well as more complex judgments such as dividing participants into basic, proficient and advanced categories.

Dr. Larsen’s five principles for test-enhanced learning in medical education

Dr Douglas Larsen
Douglas Larsen

Posted by John Kleeman

In the first part of this interview, Dr. Douglas Larsen, an expert in medical education at the Washington University in St Louis explained his research on how tests and quizzes taken during learning act as retrieval practice and aid learning and retention in medical education. Answering questions in written or computer tests gives practice in recollecting relevant facts and aids future retrieval of such facts when they are needed. In this final part of the interview, he explains his five principles to implement test-enhanced learning successfully.

Your research shows that tests in medical education can significantly help long-term retention. What advice would you give medical educators?

A big misunderstanding is that the research suggests promoting more summative tests. What we’re actually talking about here is changing how we teach people, and simply having more tests at the end of a course probably won’t change a lot. This is an opportunity for educators to think about what it is they want students to learn to do. And then to make sure that the practice of doing it is incorporated in the entire educational process and not just simply at the end.

One of the things that this research has shown is that cramming (e.g. intensive study before an exam) leads to very short-term effects. The benefits disappear quite quickly.

We have come up with 5 principles we think are important for long-term retention.

What are the 5 principles?

1. Closely align the testing with educational objectives.

2. Make sure the test involves generating or recalling, not just recognition.

Generation questions include free-recall, fill-in-blank, short-answer or essay-type questions. We’re still researching this area, but it seems the more that you force the learner to generate and organize their own structure of knowledge, the better. So the less the question enforces structure, the better.

Some studies have shown benefits of multiple-choice tests, others have shown them to be no better than studying. I think the key to the success of a question is the amount of processing required. In some multi-step multiple-choice questions, you have to process and generate an answer, not just recognize the right answer, and this is better. Retrieval and having an opportunity to organize information yourself is important rather than just picking something out of a list.

3. Adequate repetition

There need to be enough opportunities that the knowledge or skill “sinks in”.

Just like when learning the piano, you have to practice many times. When learning information, there need to be multiple practice opportunities. It seems that procedural knowledge may not need to be repeated as often as declarative facts. We’ve seen in some studies that a single testing event can have effects years later when you are dealing with procedural information.

But declarative facts go away very quickly. In one of my studies, learning of facts was measured at 60-80% initially but had dropped to 40-50% in 2 weeks. You need a lot of repetition to interrupt the forgetting curve and maintain the information. The more times you retrieve something, the more likely you are to retain it.

4. Adequate spacing

There has been research to show that if you want learning to last months and years, you need to space out your testing on the order of weeks and months, not days or hours.

5. You need to have adequate feedback.

There is definitely a testing effect without feedback, but the research has shown that with feedback, the effect is greatly amplified. People just learn more.

What is good practice in feedback?

There are a couple of principles with feedback.

One is that when people simply get immediate feedback, where someone answers a question and then immediately gets told whether it’s correct or incorrect – they probably don’t retain it as well. There needs to be a degree of delay in the feedback. The reason for that is that we need to wash the information out of short term / working memory, and give them a chance to re-process it.

Feedback that leads to re-processing is likely to be the most beneficial. Where you are forced to go back and work out why you answered incorrectly, and how that compares to the correct answer. It’s important that learners actively process the feedback, not just passively read it. For instance, one technique we use is to have students go back and grade their own test – this makes them re-process.

What is your perspective on case study questions, where you navigate through a medical scenario and answer questions on the way?

I think those are excellent in the sense that you can better approximate your desired outcomes by aligning learning objectives. If you want people to recognize elements of a case, to deduce what they need to do and so on, you obviously have to have a context for that. So case studies can be very important. As before, it is best to structure questions so that they involve recall rather than recognition.

What impact is the research on test-enhanced learning having on medical education?

Many people are very positive and very excited. The challenge is for people to understand all the implications, and to understand that we’re not talking about more standardized tests or more summative tests, but we’re really saying that people have to go back and look at how they teach, and how they incorporate retrieval practice into the longitudinal teaching experience.

That is the biggest challenge, but my hope is that, as we keep talking about it, people will catch the vision and it will have an even greater impact on how people both teach and learn.

October Webinars on remote test monitoring, mobile assessments and more

Joan PhaupPosted by Joan Phaup

With the school year now well underway, we are happy to offer some of our own learning opportunities: three Web seminars on important topics for people involved in assessment and measurement.

You can sign up for any or all of these one-hour sessions free of charge at

Here’s the line-up for October:

Thursday, October 18, at 1 p.m. Eastern Time — Integrity Anywhere: Secure Monitoring of Higher-stakes Online Tests 

Innovative technologies make it possible to take high-stakes tests using almost any webcam and computer, anywhere in the world. Online testing helps organizations increase access to their programs and assessments, but they must balance flexibility, candidate experience, logistics and security in order to ensure the integrity and value of the exams they deliver from a distance. This web seminar, co-presented by Questionmark and ProctorU,  explores the “last mile” of high-stakes test delivery and how it can meet the goals and needs of all stakeholders.

Tuesday, October 30, at 1 p.m. Eastern Time —  Creating Assessments for Mobile Delivery

Many learning professionals are considering the use of popular and inexpensive mobile devices such as smart phones and/or tablets for delivering assessment content. This seminar will show you how to use mobile assessment delivery  on its own or as part of a blended delivery strategy to give learners retrieval practice, gather their opinions or test their knowledge while they are on the move.

October 12, 24 and 31 at assorted times: Introduction to Questionmark’s Assessment Management System

This Webinar is for people who are new to Questionmark and want an overview of our assessment management technologies. A Questionmark expert will walk you through the basics of authoring, delivering and reporting on surveys, quizzes, tests and exams. It explains the key features and functions available in Questionmark OnDemand and Questionmark Perception.