Posted by Austin Fossey
The Questionmark team has just returned from the 2014 Users Conference, where we had a wonderful time showing off our latest work, discussing assessment strategies with our customers, and learning from each other in a great selection of seminars put on by both Questionmark staff and our clients.
At this year’s conference, I field tested two new presentations: Understanding Assessment Results and Principles of Psychometrics and Measurement Design. I got some great feedback from attendees so I can fine-tune them for the future, but these topics also started a lot of interesting conversations about what we as test developers would like to be doing and what we end up doing in practice.
A recurring theme of these conversations was that people felt there were occasionally aspects of their instruments that could be improved, especially in terms of capturing evidence for a measurement or supporting the validity of the results. In some cases they had an idea of what they wanted to improve, but they either did not know the test development methods they needed to apply, or they did not know how to convince their stakeholders and managers of the importance of specific initiatives. The concept of validity came up several times in these conversations—something we have touched on previously on this blog.
The ideals and realities of the assessment industry do not always align. For example, we may wish to do a construct validity study or an Angoff cut score meeting, but we may lack the resources, time, or stakeholder buy-in to engage in these activities.
I recognize how discouraging this can be for people who constantly want to improve the validity of their inferences, but I am excited to see so many people thinking critically about their assessment designs and searching for areas of improvement. Even if we cannot always implement every research study we are interested in, understanding the principles and best practices of good assessment design and interpretation can still guide our everyday work and help us to avoid invalid results. This blog is a good place to explore some of these principles, and so are Questionmark white papers and our Learning Café videos.
I look forward to continuing to work with (and learn from) our great client base throughout 2014 as we continue to advance our products. A special thanks to our attendees and presenters who joined us at the 2014 conference!