Conceptual Assessment Framework: Building the Task Model
Posted by Austin Fossey
In my previous post, I introduced the student model—one of the three sections of the Conceptual Assessment Framework (CAF) in Evidence-Centered Design (ECD). At the other end of the CAF is the task model.
The task model defines the assumptions and specifications for what a participant can do within your assessment (e.g., Design and Discovery in Educational Assessment: Evidence-Centered Design, Psychometrics, and Educational Data Mining; Mislevy, Behrens, Dicerbo, & Levy, 2012). This may include the format of the items, the format of the assessment itself, and the work products that the participant may be expected to create during the assessment.
Most importantly, the task model should be built so that the assessment tasks are appropriate for eliciting the behavior that you will use as evidence about the assessed domain. For example, if you are assessing a participant’s writing abilities, an essay item would probably be specified in your task model instead of a slew of multiple choice items.
You may have already defined pieces of your task model without even realizing it. For example, if you are using Questionmark to conduct observational assessments in the workplace, you have probably decided that the best way to gather evidence about a participant’s proficiency is to have them perform a task in a work environment. That task model may elicit behavior (and therefore evidence) that perhaps could not be captured well in other environments, for instance a traditional computer-based assessment.
In the observational assessment example below, the task model specifies that the participant has a ladder and an environment in which they can set up and climb the ladder. The task model might also specify information about the size of the ladder and the state of the ladder when the assessment begins.
The task model can also help you avoid making inappropriate assessment design decisions that might threaten the validity of your inferences about the results.
I often see test developers use more complicated item types (like drag and drop) when a multiple choice item would have been more appropriate. For example, picture an assessment about human anatomy. If you want to know if the participant can find a kidney amongst its surrounding anatomy during an operation, then you would likely build a drag and drop item. If you just want to know if a participant knows what a kidney looks like, you may want to use a multiple choice item with three to five pictures of organs from which the participant must choose.
The task model will also encompass other design and delivery decisions you will make about your assessment. For example, a time-limit, the participant’s ability to review answers, access to resources (e.g., references, calculators), and item translations might all be specified in your task model.
By specifying your task model in advance and tying your design decisions to the inferences you want to make about the participant’s results, you can ensure that your assessment instrument is built to gather the right evidence about your participants.