New Item Analysis Report in Questionmark Analytics: The Summary Page

 Posted by Jim Farrell

When I visit customers, I find that the Item Analysis report is one of the most useful reporting capabilities of Questionmark Perception. By using it, you can tell which questions are effective and which are not – and if you don’t use it, you are “running blind:” You hope your questions are good, but do not really know if they are.

Our most recent update to Questionmark OnDemand provides a new classical test theory item analysis report — one of several reports now available in Questionmark Analytics.  This report supports all question types commonly used on quizzes, tests and exams and is fully scalable for application to large pool of participants. Let’s take a look at the report!

ItemAnalysisReport1

This is the summary page. The graph show the performance of questions in relation to one another in terms of their difficulty (p-value) and discrimination (item-total correlation). The p-value is a number from 0 to 1, and represents the proportion of people who correctly answer the question. So a question with p-value of 0.5 means that half the participants get it right and half wrong.  And a question with p-value of 0.9 means that 90% of respondents get it right.

A rule of thumb is that it’s often useful to use questions with p-value that are reasonably close to the pass score of the assessment.  For instance, if your pass score is 60%, then questions with a p-value of around 0.6 will give you good information about your participants.  However a very high or very low p-value does not give you much information about a person who answers it. If the purpose of the test is to measure someone’s knowledge or skills, then you will get more information from a question with medium p-value. Using the item analysis report is an easy way to get p-values.

The other key statistic in the item analysis report is the item-total correlation discrimination, which shows the correlation between the question score and the assessment score. Higher positive correlation values indicate that participants who obtain high question scores also obtain high assessment scores. Conversely, participants who obtain low question scores also obtain low assessment scores. Low values for questions here could indicate unhelpful questions and are worth drilling down on.

Below the graph is a table that shows some high-level details of each question composing the assessment. The table can be sorted by any of the columns. By clicking on a row/question the user goes to the detail page, which we will discuss in our next blog post on this subject.

If you are running a medium or high stakes assessment that has to be legally defensible, then you cannot confirm that the assessment is valid if you are not running item analysis. And for all quizzes, tests and exams, running an item analysis report will give you information to help you make the assessment better.

2 Responses to “New Item Analysis Report in Questionmark Analytics: The Summary Page”

  1. Sami Besalel says:

    Jim,

    This is exciting news.

    We are former On-Demand customers but now have our own OnPremise installation – which means we are 3 – 6 months behind having the opportunity to use these.

    We are just now learning to use the existing Item Analysis Report, and I’m wondering how long you will leave the old reports in place, side by side with the new ones.

    The current item analysis report indicates separate measurs for discrimination (with QM considering .15 or lower to be low) and item total correlation (with QM recommending over .19 to get above the low threshold). This summary seems to include them. Will separate statistics be provided?

    Also, do you plan to include measures of kurtosis and skew into this report in the future? Or is there another item analysis report that will allow us to use these measures? We can also do distracter analysis now based on the individual performance results in the item analysis report. Will we still have access to this data?

    Thanks, and we are excited to see Analytics grown and expand.

    Sami Besalel, Beacon Associates

  2. […] a recent post I explained the Summary page of the new Item Analysis report in Questionmark Analytics, now […]

Leave a Reply