Psychometrics and Measurement Design: A conversation
Posted by Joan Phaup
Many delegates to the Questionmark 2014 Users Conference in San Antonio March 4 – 7 want to learn about assessment-related best practices.
Austin had just joined Questionmark when he attended the 2013 conference. This time around, he’ll be more actively involved in the program, so I wanted to learn more about him and his presentation plans.
What made you decide to study psychometrics?
I was working in customer service at a certification testing company. They always brought in psychometricians to build their assessments. I’d never heard of psychometrics before, but I had studied applied math as an undergraduate and thought the math behind psychometrics was interesting. I liked the idea of doing analytical work and heard that psychometricians are always in demand, so I got started right away studying educational measurement at the University of Maryland.
How do you make principles of psychometrics understandable to, well, mere mortals?
I don’t think psychometricians are different than anybody else. Most of it is applying a probabilistic model to a set of data to make an inference about an unobserved trait. Those models are based on concepts or theories, so you don’t have to explain the math as long as you can explain the theory. People understand that.
I really like evidence-centered design, because it provides principles and a vocabulary that can be used by everyone involved in assessments. Using this framework, psychometricians can communicate about measurement design with subject matter experts, item writers, curriculum specialists, programmers, policy makers — all the stakeholders, from start to finish.
Who do you think would benefit from attending your presentation about psychometrics and measurement design?
People who feel they are applying the same test development formula day in and day out and who wonder if there might be a better way to do it. Even with certifications, which usually follow excellent standards based on best practices, we should always be critical about our assessments and we should always be aggressive about ensuring validity. It would be great to see people there who want to be mindful of every decision they make in assessment design.
How could people prepare for this session?
I hope they bring examples of their own test development process and validity studies. We can discuss people’s own experiences and the hurdles they have faced with their measurement design. Other than that I would say just bring an open mind.
What would you like your audience to take away from your presentation?
People who may be new to measurement design and psychometric concepts like validity can take away some tools to use in their assessment programs. I hope that if more experienced people come, they can learn from each others’ experiences and go away with new ideas about their own approach to assessment design.
What do you hope to take away from the Users Conference?
I want to harvest a lot of feedback with our clients during conversations and focus groups, so that we can recalibrate ourselves for the work we are doing and prioritize our tasks.