Question Type Report: Use Cases

Austin Fossey-42Posted by Austin Fossey

A client recently asked me if there is a way to count the number of each type of item in their item bank, so I pointed them toward the Question Type Report in Questionmark Analytics. While this type of frequency data can also be easily pulled using our Results API, it can be useful to have a quick overview of the number of items (split out by item type) in the item bank.

The Question Type Report does not need to be run frequently (and Analytics usage stats reflect that observation), but the data can help indicate the robustness of an item bank.

This report is most valuable in situations involving topics for a specific assessment or set of related assessments. While it might be nice to know that we have a total of 15,000 multiple choice (MC) items in the item bank, these counts are trivial unless we have a system-wide practical application—for example planning a full program translation or selling content to a partner.

This report can provide a quick profile of the population of the item bank or a topic when needed, though more detailed item tracking by status, topic, metatags, item type, and exposure is advisable for anyone managing a large-scale item development project. Below are some potential use cases for this simple report.

Test Development and Maintenance:
The Question Type Report’s value is primarily its ability to count the number of each type of item within a topic. If we know we have 80 MC items in a topic for a new assessment, and they all need be reviewed by a bias committee, then we can plan accordingly.

Form Building:
If we are equating multiple forms using a common-item design, the report can help us determine how many items go on each form and the degree to which the forms can overlap. Even if we only have one form, knowing the number of items can help a test developer check that enough items are available to match the blueprint.

Item Development:
If the report indicates that there are plenty of MC items ready for future publications, but we only have a handful of essay items to cover our existing assessment form, then we might instruct item writers to focus on developing new essay questions for the next publication of the assessment.

Question type

Example of a Question Type Report showing the frequency distribution by item type.

 

Using Questionmark’s OData API to Create a Response Matrix

Austin FosseyPosted by Austin Fossey

A response matrix is a table of data in which each row represents a participant’s assessment attempt and each column represents an item. The cells show the score that each participant received for each item – valuable information that can help you with psychometric analysis.

The Questionmark OData API enables you to create this and other custom data files by giving you flexible, direct access to raw item-level response data.

You can already see participant’s item-level response data in Questionmark reports, but the Questionmark reports group data together for one assessment at a time.

If you have a large-scale assessment design with multiple equated forms, you may want to generate a matrix that shows response data for common items that are used across the forms.

The example below shows a response matrix created with OData in Microsoft Excel 2013 using the PowerPivot add-in. The cells in a response matrix are coded with the score that the participant received for each item (e.g., 1 = correct and 0 = incorrect). (If an item was not delivered to a participant, the cell will be returned blank, though you can impute other values as needed.)

matrix

You can use OData to create a response matrix that can be used for form equating or as input files for item calibration in Item Response Theory (IRT) software. These data are also helpful if you want to check a basic item-level calculation, like the p-value for the item across all assessments. (Note that item-total correlations can only be calculated if the total score has been equated for all forms.)

Visit Questionmark’s website for more information about the OData API. (If you are a Questionmark software support plan customer, you can get step-by-step instructions for using OData to create a response matrix in the Premium Videos section of the Questionmark Learning Café.)

Using Questionmark’s OData API to Analyze Item Key Distribution

Austin FosseyPosted by Austin Fossey

The Questionmark OData API, which offers flexible access to data for the creation of custom reports, can help you ensure the quality of your tests.

For instance, you can use OData to create a frequency table of item keys in a multiple choice assessment. This report shows the number of items that have the first choice as the correct answer, the number of items that have the second choice as the correct answer, et cetera.

analyze 1

Why do we care about how often each choice number is the item key? If there is a pattern in how correct choices are assigned, it may affect how participants perform on the test, and this can lead to construct-irrelevant variance; i.e., the scores are being affected by factors other than the participant’s knowledge and abilities.

Let’s say that our assessment has 50 items, and on 30 (60%) of those items the second choice is the item key. Now let’s put ourselves in the shoes of a qualified participant. Halfway through the assessment, we might start thinking, “Gosh, I just picked the second choice four times in a row. Maybe I should go back and check some of those answers.” Because of poor test design, we are second-guessing our answers. Even if we do not change our responses, time is being wasted and test anxiety is rising, which might negatively affect our responses later in the assessment.

The opposite problem may arise too. If an unqualified participant figures out that the second choice is most often the key, he or she may pick the second choice even when he or she does not know the answer, resulting in an inflated score.

When looking at the distribution of keys across a selected response assessment, we expect to see an even distribution of the keys across the choices. For example, if we have a multiple choice assessment with four choices in each item labeled A, B, C, and D, we would like to see the 25% of the keys assigned to each of these choices.

analyze 2

You do not have to limit your assessment research to this example. The beauty of OData is that you can access your data whenever you have a new question you would like to investigate. For example, instead of reviewing the frequencies of your keys, you may want to determine the ratio of the length of the key to the length of the other options (a common item writing mistake is to write keys that are noticeably longer than the distractors). You may also want to look for patterns in the keys’ text (e.g., 10 items all have “OData” as the correct choice).

Click here for more information about the OData API. (If you are a Questionmark software support plan customer, you can get step-by-step instructions for using OData to create the item key frequency table in the Premium Videos section of the Questionmark Learning Café.)

OData Tutorials with Excel PowerPivot

Austin FosseyPosted by Austin Fossey

In case you missed this earlier post, Questionmark has an OData feed that allows users to get direct access to data in their Results Warehouse—the same data source that is used to drive the reports in Questionmark Analytics.

If you have not done so, I encourage you to check out our OData API for Analytics video to learn how to connect to the OData feed, import data and create a simple report.

There are three major benefits of having this access to the Results Warehouse data:

  1. You can grab your data anytime you wish to explore a new research question about your assessment results
  2. You can manipulate those data and run your own analyses or transformations in any way you see fit
  3. You can feed those data into your own reporting tools or many other programs that already consume OData

Of course, the only reason we create assessments and collect data is so that we can make informed decisions, and the data in the OData feed just come across in their raw state.

To help you make sense of the data, we are developing short video tutorials that walk through introductory OData examples using the free PowerPivot add on for Excel.

These tutorials will cover the following topics:

  • Creating a response matrix with OData
  • Using OData to analyze the distribution of correct choices in the items on an assessment
  • Calculating a new variable in PowerPivot using the data from the OData feed
  • Using OData to generate a list of scores for participants who took multiple forms of an assessment

We will announce these new videos in the blog in the coming weeks, and you will be able to find them in the Questionmark Learning Café.

Odata 1

If you have a research question that you think could be addressed with OData, please let us know! Your suggestion may lead to additional tutorials that will help other people expand their own research.