Item Development – Managing the Process for Large-Scale Assessments

Austin FosseyPosted by Austin Fossey

Whether you work with low-stakes assessments, small-scale classroom assessments or large-scale, high-stakes assessment, understanding and applying some basic principles of item development will greatly enhance the quality of your results.

This is the first in a series of posts setting out item development steps that will help you create defensible assessments. Although I’ll be addressing the requirements of large-scale, high-stakes testing, the fundamental considerations apply to any assessment.

You can find previous posts here about item development including how to write items, review items, increase complexity, and avoid bias. This series will review some of what’s come before, but it will also explore new territory. For instance, I’ll discuss how to organize and execute different steps in item development with subject matter experts. I’ll also explain how to collect information that will support the validity of the results and the legal defensibility of the assessment.

In this series, I’ll take a look at:

Item Dev.

These are common steps (adapted from Crocker and Algina’s Introduction to Classical and Modern Test Theory) taken to create the content for an assessment. Each step requires careful planning, implementation, and documentation, especially for high-stakes assessments.

This looks like a lot of steps, but item development is just one slice of assessment development. Before item development can even begin, there’s plenty of work to do!

In their article, Design and Discovery in Educational Assessment: Evidence-Centered Design, Psychometrics, and Educational Data Mining, Mislevy, Behrens, Dicerbo, and Levy provide an overview of Evidence-Centered Design (ECD). In ECD, test developers must define the purpose of the assessment, conduct a domain analysis, model the domain, and define the conceptual assessment framework before beginning assessment assembly, which includes item development.

Once we’ve completed these preparations, we are ready to begin item development. In the next post, I will discuss considerations for training our item writers and item reviewers.

Assessment Report Design: Reporting Multiple Chunks of Information

Austin FosseyPosted by Austin Fossey

We have discussed aspects of report design in previous posts, but I was recently asked whether an assessment report should report just one thing or multiple pieces of information. My response is that it depends on the intended use of the assessment results, but in general, I find that a reporting tool is more useful for a stakeholder if it can report multiple things at once.

This is not to say that more data are always better. A report that is cluttered or that has too much information will be difficult to interpret, and users may not be able to fish out the data they need from the display. Many researchers recommend keeping simple, clean layouts for reports while efficiently displaying relevant information to the user (e.g., Goodman & Hambleton, 2004; Wainer, 1984).

But what information is relevant? Again, it will depend on the user and the use case for the assessment, but consider the types of data we have for an assessment. We have information about the participants, information about the administration, information about the content, and information about performance (e.g., scores). These data dimensions can each provide different paths of inquiry for someone making inferences about the assessment results.

There are times when we may only care about one facet of this datascape, but these data provide context for each other, and understanding that context provides a richer interpretation.

Hattie (2009) recommended that a report should have a major theme; that theme should be emphasized with between five to nine “chunks” of information. He also recommended
that the user have control of the report to be able to explore the data as desired.

Consider the Questionmark Analytics Score List Report: Assessment Results View. The major theme for the report is to communicate the scores of multiple participants. The report arguably contains five primary chunks of information: aggregate scores for groups of participants, aggregate score bands for groups of participants, scores for individual participants, score bands for individual participants, and information about the administration of the assessment to individual participants.

Through design elements and onscreen tools that give the user the ability to explore the data, this report with five chunks of information can provide context for each participant’s score. The user can sort participants to find the high- and low-performing participants, compare a participant to the entire sample of participants, or compare the participant to their group’s performance. The user can also compare the performance of groups of participants to see if certain groups are performing better than others.

rep 1

Assessment Results View in the Questionmark Analytics Score List Report

Online reporting also makes it easy to let users navigate between related reports, thus expanding the power of the reporting system. In the Score List Report, the user can quickly jump from Assessment Results to Topic Results or Item Results to make comparisons at different levels of the content. Similar functionality exists in the Questionmark Analytics Item Analysis Report, which allows the user to navigate directly from a Summary View comparing item statistics for different items to an Item Detail view that provides a more granular look at item performance through interpretive text and an option analysis table.

Nine tips on recommended assessment practice — from Barcelona

John Kleeman HeadshotPosted by John Kleeman

Something I enjoy most about our users conferences is the chance to learn from experts about good practice in assessments. Most of our customers have deep knowledge and insightful practical experience, so there is always much to learn.

Here are some tips I picked up last week at our recent European Users Conference in Barcelona.Questionmark2013_DSC3209

1. Make sure to blueprint. It’s critical to have a detailed design (often called a blueprint) for an assessment – or as one user shared, “Without a blueprint, you don’t have an assessment”.

2. Network to get SMEs. With technology changing quickly, if your assessments assess IT or other new technology, the content changes very quickly and the quality of your subject matter experts (SMEs) who create and review items is critical. As an assessment owner, use networking skills to get the right SMEs on board; getting them engaged and building trust are essential.

3. Test above knowledge. Develop questions that test application or comprehension, for instance using scenarios. They are more likely to make your test valid than questions that simply test facts.

4. Give employees ownership of their own compliance testing. If employees have to take annual refresher tests, give them the responsibility to do so and encourage sel- learning and pre-reading. Give them plenty of time (e.g. 6 weeks’ warning), but make it their responsibility to take and pass the test in the window, not yours to keep on reminding them.

5. Gather feedback from participants. Make sure you solicit feedback from your participants on tests and the testing experience. That way you will learn about weak questions and how to improve your testing process. And you also make participants feel that the process is fairer.

6. Use job/task analysis. Asking questions about jobs and tasks is the best way to specify the criteria used to judge competency or proficiency. These questions can be automated in Questionmark right now. Watch this space for improvements coming to make this easier.

7. Look at Questionmark Live for item review workshops. If you have any informal or informal process for having groups of people working on or reviewing items, look at Questionmark Live. It’s free to use, has great group working capability and improves productivity. A lot of organizations are having success with it.

8. Keep feedback short and to the point… especially on mobile devices where people won’t read long messages.Questionmark2013_DSC3215

9. Look for live data, not just your rear view mirror. Data is important – without measurement we cannot improve. But make sure the data you are looking at is not dead data. Looking into the rear view mirror of what happened in the past doesn’t help as much as using reports and analytics from Questionmark to discover what is happening now, and use that data to improve things.

I hope some of these tips can help you in your work with assessments.

I will write in the spring about the tips I gather at the 2014 U.S. Users Conference in San Antonio!

OData Tutorials with Excel PowerPivot

Austin FosseyPosted by Austin Fossey

In case you missed this earlier post, Questionmark has an OData feed that allows users to get direct access to data in their Results Warehouse—the same data source that is used to drive the reports in Questionmark Analytics.

If you have not done so, I encourage you to check out our OData API for Analytics video to learn how to connect to the OData feed, import data and create a simple report.

There are three major benefits of having this access to the Results Warehouse data:

  1. You can grab your data anytime you wish to explore a new research question about your assessment results
  2. You can manipulate those data and run your own analyses or transformations in any way you see fit
  3. You can feed those data into your own reporting tools or many other programs that already consume OData

Of course, the only reason we create assessments and collect data is so that we can make informed decisions, and the data in the OData feed just come across in their raw state.

To help you make sense of the data, we are developing short video tutorials that walk through introductory OData examples using the free PowerPivot add on for Excel.

These tutorials will cover the following topics:

  • Creating a response matrix with OData
  • Using OData to analyze the distribution of correct choices in the items on an assessment
  • Calculating a new variable in PowerPivot using the data from the OData feed
  • Using OData to generate a list of scores for participants who took multiple forms of an assessment

We will announce these new videos in the blog in the coming weeks, and you will be able to find them in the Questionmark Learning Café.

Odata 1

If you have a research question that you think could be addressed with OData, please let us know! Your suggestion may lead to additional tutorials that will help other people expand their own research.

Welcome to Getting Results—The Questionmark Blog!

joan-small1Posted by Joan Phaup

We have always believed in promoting best practices in assessment and enjoy lively dialog about the important role assessments play in measuring learning and improving performance. This blog gives us yet another way to keep the conversation going!

Here you will find news about learning events, products, and trends in learning and assessment. We will also post technical pointers, case studies and advice about effective item writing, secure delivery, reporting, analyzing results and other essentials. Check in here for links to learning resources including white papers and podcast interviews with assessment professionals and experts on best practices.

Stay connected with us and with the wider testing and assessment community right here. Ask questions, post comments and take the opportunity to spark discussions.

Together, let’s explore how best to create, deliver and report on assessments that help individuals and organizations work more effectively. And let’s have some fun in the process!

We’ll update the blog often, and you can stay in touch easily by subscribing to this blog through our RSS feed. We hope you will contribute to the blog, too, by submitting your comments and sharing any particular post you like!