Question Type Report: Use Cases

Austin Fossey-42Posted by Austin Fossey

A client recently asked me if there is a way to count the number of each type of item in their item bank, so I pointed them toward the Question Type Report in Questionmark Analytics. While this type of frequency data can also be easily pulled using our Results API, it can be useful to have a quick overview of the number of items (split out by item type) in the item bank.

The Question Type Report does not need to be run frequently (and Analytics usage stats reflect that observation), but the data can help indicate the robustness of an item bank.

This report is most valuable in situations involving topics for a specific assessment or set of related assessments. While it might be nice to know that we have a total of 15,000 multiple choice (MC) items in the item bank, these counts are trivial unless we have a system-wide practical application—for example planning a full program translation or selling content to a partner.

This report can provide a quick profile of the population of the item bank or a topic when needed, though more detailed item tracking by status, topic, metatags, item type, and exposure is advisable for anyone managing a large-scale item development project. Below are some potential use cases for this simple report.

Test Development and Maintenance:
The Question Type Report’s value is primarily its ability to count the number of each type of item within a topic. If we know we have 80 MC items in a topic for a new assessment, and they all need be reviewed by a bias committee, then we can plan accordingly.

Form Building:
If we are equating multiple forms using a common-item design, the report can help us determine how many items go on each form and the degree to which the forms can overlap. Even if we only have one form, knowing the number of items can help a test developer check that enough items are available to match the blueprint.

Item Development:
If the report indicates that there are plenty of MC items ready for future publications, but we only have a handful of essay items to cover our existing assessment form, then we might instruct item writers to focus on developing new essay questions for the next publication of the assessment.

Question type

Example of a Question Type Report showing the frequency distribution by item type.

 

Using the Question Type Report in Questionmark Analytics

This week’s “how to” article highlights the “Question Type Report,” one of the Questionmark Analytics reports now available in Questionmark OnDemand.

  • What it does: This item bank report tells how many questions you have in your repository by question type (e.g., multiple choice, yes/no, etc.).
  • Who should use it: This report gives testing, assessment, learning and education professionals a quick view of what their item bank looks like.
  • How it looks: This report lists various color-coded question types (e.g., numeric, essay, multiple response, multiple choice) and  graphically represents how many of each type there are. The report can be viewed in a web browser or downloaded and distributed as a PDF. The CSV version of this report lists question types in the first column and the number of questions per topic in the remaining columns. The question detail CSV distribution provides information such as each question’s Perception question ID, status, topic and question type. You can see the information for your entire repository or just for specific topics.