Posted by Greg Pope

In my previous blog post I highlighted some of the essential things to look for in a typical Item Analysis Report. Now I will dive into the nitty-gritty of item analysis, looking at example questions and explaining how to use the Questionmark Item Analysis Report in an applied context for a State Capitals Exam.

The Questionmark Item Analysis Report first produces an overview of question performance both in terms of the difficulty of questions and in terms of the discrimination of questions (upper minus lower groups). These overview charts give you a “bird’s eye view” of how the questions composing an assessment perform. In the example below we see that we have a range of questions in terms of their difficulty (“Item Difficulty Level Histogram”), with some harder questions (the bars on the left), most average-difficulty questions (bars in the middle), and some easier questions (the bars on the right). In terms of discrimination (“Discrimination Indices Histogram”) we see that we have many questions that have high discrimination as evidenced by the bars being pushed up to the right (more questions on the assessment have higher discrimination statistics).

Overall, if I were building a typical criterion-referenced assessment with a pass score around 50% I would be quite happy with this picture. We have more questions functioning at the pass score point with a range of questions surrounding it and lots of highly discriminating questions. We do have one rogue question on the far left with a very low discrimination index, which we need to look at.

The next step is to drill down into each question to ensure that each question performs as it should. Let’s look at two questions from this assessment, one question that performs well and one question that does not perform so well.

The question below is an example of a question that performs nicely. Here are some reasons why:

• Going from left to right, first we see that the “Number of Results” is 175, which is a nice sample of participants to evaluate the psychometric performance of this question.
• Next we see thateveryone answered the question (“Number not Answered” = 0), which means there probably wasn’t a problem with people not finishing or finding the questions confusing and giving up.
• The “P Value Proportion Correct” shows us that this question is just above the pass score where 61% of participants ‘got it right.’ Nothing wrong with that: the question is neither too easy nor too hard.
• The “Item Discrimination” indicates good discrimination, with the difference between the upper and lower group in terms of the proportion selecting the correct answer of ‘Salem’ at 48%. This means that of the participants with high overall exam scores, 88% selected the correct answer versus only 40% of the participants with the lowest overall exam scores. This is a nice, expected pattern.
• The “Item Total Correlation” backs the Item Discrimination up with a strong value of 0.40. This means that of all participants who answered the questions, the pattern of high scorers getting the question right more than low scorers holds true.
• Finally we look at the Outcome information to see how the distracters perform. We find that each distracter pulled some participants, with ‘Portland’ pulling the most participants, especially from the “Lower Group.” This pattern makes sense because those with poor state capital knowledge may make the common mistake of selecting Portland as the capital of Oregon.

The psychometricians, SMEs, and test developers reviewing this question all have smiles on their faces when they see the item analysis for this item.

Next we look at that rogue question that does not perform so well in terms of discrimination-–the one we saw in the Discrimination Indices Histogram. When we look into the question we understand why it was flagged:

• Going from left to right, first we see that the “Number of Results” is 175, which is again a nice sample size: nothing wrong here.
• Next we see everyone answered the question, which is good.
• The first red flag comes from the “P Value Proportion Correct” as this question is quite difficult (only 35% of participants selected the correct answer). This is not in and of itself a bad thing so we can keep this in memory as we move on,
• The “Item Discrimination” indicates a major problem, a negative discrimination value. This means that participants with the lowest exam scores selected the correct answer more than participants with the highest exam scores. This is not the expected pattern we are looking for: Houston, this question has a problem!
• The “Item Total Correlation” backs up the Item Discrimination with a high negative value.
• To find out more about what is going on we delve into the Outcome information area to see how the distracters perform. We find that the keyed-correct answer of Nampa is not showing the expected pattern of upper minus lower proportions. We do, however, find that the distracter “Boise” is showing the expected pattern of the Upper Group (86%) selecting this response option much more than the Lower Group (15%). Wait a second…I think I know what is wrong with this one, it has been mis-keyed! Someone accidently assigned a score of 1 to Nampa rather than Boise.

No problem: the administrator pulls the data into the Results Management System (RMS), changes the keyed correct answer to Boise, and presto, we now have defensible statistics that we can work with for this question.

The psychometricians, SMEs, and test developers reviewing this question had a frown on their faces at first but those frowns were turned upside down when they realized it is just a simple mis-keyed question.

In my next blog post I would like share some observations on the relationship between Outcome Discrimination and Outcome Correlation.

Are you ready for some light relief after pondering all these statistics? Then have some fun with our own State Capitals Quiz.

Posted by Joan Phaup

The suspense is over for those of you waiting for Part 2 of our tutorial on creating and editing assessments in Questionmark Perception! This video demonstrates how to customize and control how your assessment works. Find out how to use the assessment editor to arrange time limits, security options and other settings. Learn how to establish  the look and feel of your assessment, create  jump blocks, set up feedback options, organize post-assessment email messages and perform many other tasks. 

Click here to review Part 1 of the assessment creation tutorial, and here to watch our video about creating and managing questions.

Posted by Joan Phaup

Earlier this month I shared a tutorial on creating and managing questions using Questionmark Perception. Now it’s time to move on to organizing your questions into  assessments. This video from the Questionmark Learning Cafe will show you how to create assessment folders and assign administrators to them. It will demonstrate how Perception’s assessment wizard guides you step-by-step through the process of creating surveys, quizzes, tests and exams.

What type of assessment do you want? Do you want to set a time limit for taking it?  Do you want to provide feedback to participants? How do you want to order your questions, and how many questions would like to include? Do you want to set a pass/fail threshold? The wizard will walk you through these and many other decisions as you create your assessment.

In my next post I will share another video showing how to customize your assessments and control how they work. Watch this space for Part 2!

Posted by John Kleeman

I’m reporting from the E-Assessment Live event at Loughborough University on a practical experience of crowdsourcing assessment content organized by our events team. We had a session with around 20 workstations in a room and gave everyone access via a browser to Questionmark Live, our new software-as-a-service authoring system that allows anyone with a browser to create questions easily and email them out for use in Questionmark Perception.

Most of the people in the room  were not familiar with Questionmark. We asked them all to create a question and email them to me from the system. They all logged into Questionmark Live and wrote a question on their home town which I brought into Questionmark Perception very easily, and within 20 minutes from the first question being authored we had an assessment. See below for a screenshot.

I think the availability of applications like Questionmark Live, which allow easy creation of questions by lots of people at the same time and amalgamation into an assessment, is going to make a big difference in the assessment world. Obvious ideas include getting students to create questions for each other and having SMEs brainstorm and then review questions as a group in an item writing workshop. Essentially harness the power of the crowd by letting each person contribute simultaneously rather than write items sequentially or hierarchically.

I am sure there will be ways of using crowdsourcing for questions that no one has thought of yet and this will hugely improve our productivity. Questionmark Live is free to Questionmark software support plan customers and open for anyone to evaluate. Seeing is believing, so I encourage you to try it out on our website.

Posted by Joan Phaup

Questions are the building blocks for all sorts of assessments, from surveys and quizzes to tests and exams. Whether you create items within Perception or import them from other programs, organizing them by topics helps you manage item banks effectively. Topics can be aligned with learning objectives — a big advantage in identifying knowledge gaps, measuring skills and prescribing learning and development plans.

The following feature tour  shows how to set up a topic structure and then add questions to it. It shows how to edit questions, add graphics and fine-tune the layout before previewing and saving questions. If you are not familiar with Questionmark Perception, this video will give you a basic understanding of how to create questions and put together a well-organized assessment.

You will find other simulations in Questionmark’s Learning Cafe, which includes resources about best practices as well as tutorials about the Questionmark Perception assessment management system.

Posted by Greg Pope

In my last post I talked a bit about Classical Test Theory (CTT) to lay the foundation for a discussion of item analysis analytics using CTT. In this post I will talk about the high-level purpose and process of conducting an item analysis. The general purpose of conducting an item analysis is to find out whether the questions composing an assessment are performing in a manner that is psychometrically appropriate and defensible. Item analyses are used to evaluate the psychometric performance of questions. They help us find out whether items need to be improved (sent back to development), sent to the scrap heap, or left as they are because they meet all the criteria for being included in an assessment.

I’d like to share a tip about how some of my colleagues decide whether to revise a problematic looking question or throw it away as “unfixable.”  This involves setting a review time limit for each question that needs to be reviewed. In an item analysis review meeting which may involve psychometricians, subject matter experts, exam developers and other stakeholders, each question could be reviewed for no more than a pre-determined period of time, say 10 minutes. If an effective revision for the question does not become apparent within that period of time, the question goes to the scrap bin and a new question is developed by SMEs to take its place.

Many organizations beta test questions in order to choose those that should be included in an actual assessment. Questionmark Perception offers the delivery status field of “Experimental,” which allows beta questions to be included/interspersed within an actual assessment form but not scored and therefore not counted as part of the calculation of participant assessment scores. More on the topic of beta testing another time though…

In my next post I will discuss some essential things to look for in an Item Analysis Report.

Posted by Eric Shepherd

I have been watching the Questionmark Blog with interest and thought that, as Questionmark’s CEO, it was about time that I made a contribution!

The Questionmark Blog was started to keep you in touch with our products, our news releases, learning materials and our Product Owners’ points of view.  We’ve been focusing on articles that assist assessment practitioners and instructional designers; recently we previewed how embedding syndicated assessments within wikis, web pages and blogs can support the learning process.

Separate to this initiative I have been running a personal blog (http://blog.eric.info) to bring you more abstract thoughts, observations from travels, and distillations of conversations that I’ve enjoyed along the way.  Not surprisingly the Tag Cloud quickly shows what I blog about, Assessments, Books, Travel and Questionmark.  Here are some links that you might find interesting:

•    Recent article on Learning Environments that explains how systems are now being built around Single Sign-on Portals, Wikis, Blogs and Data Warehouses
•    Questionmark Live – Story Behind the Story
•    Assessments Fundamentals with articles on Fidelity of an Assessment, Blooms Taxonomy, Item Analysis,   Types Of Assessments (Formative, Diagnostic, Summative, and Surveys), and many more.
•    A couple of YouTube videos, one titled Assessment as they relate to Learning Professionals
•    My Favorite Books , which relate to mostly to best practices in management and assessments. I’ll be posting more as I get time.

I look forward to meeting you out in the web 2.0 world!

Posted by Jim Farrell

Jay Cross recently mentioned in his Learning Blog a Harvard Business blog post by John Hagel III, John Seely Brown, and Lang Davison about The Collaboration Curve.  I was struck by this statement from their article:

…the more participants–and interactions between those participants–you add to a carefully designed and nurtured environment, the more the rate of performance improvement goes up

This is why Questionmark Live is so important. Think of it as crowdsourcing content from participants who may previously  have been indifferent to the assessment process within your organization. As participation and collaboration increases, the quality of your questions will improve and become far more job-relevant. Look at the following quote from our friends at the Harvard School of Business and replace collaboration curves with Questionmark Live.

Collaboration curves hold the potential to mobilize larger and more diverse groups of participants to innovate and create new value.

Questionmark Live is a vital tool in the creation of a true learnscape.

Posted by Joan Phaup

Writing effective questions takes time and practice. Whether your goal is to measure knowledge and skills, survey opinions and attitudes or enhance a learning experience, poorly worded questions can adversely affect the quality of the results.

I’ve gleaned the following tips for writing and reviewing questions from Questionmark’s learning resources:

1. Keep stems and statements as short as possible and use clear, concise language.
2. Use questions whenever possible (What, Who, When, Where, Why and How).
3. Maintain grammatical consistency to avoid cueing.
4. List choices in a logical order.
5. Avoid negatives, especially double negatives.
6. Avoid unnecessary modifiers, especially absolutes (e.g. always, never, etc.).
7. Avoid “All of the above” and use of “None of the above” with caution.
8. Avoid vague pronouns (e.g. it, they).
9. Avoid conflicting alternatives.
10. Avoid syllogistic reasoning choices (e.g. “both a and b are correct”) unless absolutely necessary.
11. Avoid providing cues to correct answer in the stem.
12. Avoid providing clues to the answer of one question in another question.

If you would like more information about writing question and assessments, a good place to start is the Questionmark Learning Cafe.