How to create reliable tests using JTA
Posted by Jim Farrell
The gold standard of testing is to have valid test results. You must always be asking yourself: Does this test really test what it is supposed to test? Are the topics covered going to tell me whether or not the participant has the knowledge or skills to perform the tasks required for the job? The only way to be 100 percent sure is to truly know what the tasks are, how important they are, and how often they are performed to make sure you are asking relevant questions. All of this information is covered in a Job Task Analysis (JTA). (A JTA question type is available in Questionmark Live).
A JTA is an exercise that helps you define the tasks a person in a particular position needs to perform or supervise and then measure the:
1. difficulty of the task
2. importance of the task
3. frequency of the task
Together, these dimensions are often called the DIF. There may be other dimensions you may want to measure but the DIF can help you build a competency model for the job. A competency model is a visual representation of the skills and knowledge a person needs to be highly successful. This is created by interviewing subject matter experts (SMEs) who define the DIF for each task. This sounds like a piece of cake, right? Well it can be, but many people often disregard creating a JTA because of the time and expense. The thought of going out and interviewing SMEs and then going back and correlating a ton of data sounds daunting. That is where Questionmark can help out.
With our JTA question type, you can create a list of tasks and dimensions to measure them. You can then send out the survey to all of your SMEs and then use specific job task analysis reports to vet and create your competency model. Now that makes it a piece of cake!
Let’s take a quick look at the process a little more closely. In authoring, you can define your tasks and dimensions by entering them directly or importing them from an outside source.
The final step of the process is running reports broken down by different demographic properties. This will give you the opportunity to sit down and analyze your results, vet them with your SMEs, and develop your competency model.
Let’s get to why we are here…designing a test that will yield valid, meaningful results. Now that you know what needs to be tested, you can create a test blueprint or specification. This documentation will drive your item development process and make sure you have the right questions because you can map them back to the tasks in your competency model.