Understanding Assessment Validity: Criterion Validity


Posted by Greg Pope

In my last post I discussed three of the traditionally defined types of validity: criterion-related, content-related, and construct-related. Now I will talk about how your organization could undertake a study to investigate and demonstrate criterion-related validity.

So just to recap, criterion-related validity deals with whether assessment scores obtained for participants are predictive of something related to the goal of the assessment. For example, if a training program conducts a four-day sales training course, at the end of which an exam is administered designed to measure trainees’ knowledge and skills in the area of product sales, one may wonder whether the exam results have any relationship with actual sales performance. If the sales course exam scores are found to be related to/predict “real world” sales performance to a high degree, then we can say that there is a high degree of criterion-related validity between the intermediate variable (sales course exam scores) and the final or ultimate variable (sales performance).

So how does one find out whether high scores on the sales course exam correspond to high sales performance (and whether low scores on the sales course exam correspond to low sales performance)? Well, within an organization there may be some “feeling” about this, for example instructors seeing star students in the course bring in big sales numbers, but how do we get some hard numbers to back this up? You will be glad to hear that you don’t need a supercomputer and a room full of PhDs to figure this out! All you need to get some data on this are some good assessment results and some corresponding sales numbers for people who have gone through the course.

The first step is to gather the sales course exam scores for the participants who took the exam. In Questionmark Perception you can use the Export to ASCII or Export to Excel reports to output in a nice user-friendly format the assessment scores for the participants who took the sales course exam. Next you will want to match the participants for whom you have exam scores with their sales numbers (e.g., how much has each salesperson sold in the last 3 months). You may want to wait a few months after these participants have taken the exam and have been out in the field selling for a while, or you could look at historical sales data if you have it. Now you put this data together into an Excel spreadsheet (or SPSS or other analysis tool if you are savvy with those tools) to analyze in way similar to this:

validity 2Next you may want to produce a scatter plot and conduct a correlation and trend line between sales course exam scores and sales dollars for the last three months:

validity 5 correct

We find the correlation is 0.901, which is very high positive relationship (people with higher sales course exam scores bring in more sales dollars). This would suggest a high degree of criterion-related validity in that the sales course exam scores do indeed predict sales performance.

To go one step further, you can take the equation produced in Excel included on the scatter plot trend line and for new sales people taking the sales course exam you can predict how much sales revenue they might bring in: y = 21049x – 3366.2 (y=estimated sales performance in dollars, x= sales course exam score). Suppose a new sales person (Rick Thomas) obtains a sales course exam score of 73%. Just plug this into the equation and y=21049(0.73)-3366.2 = $11,999.57. Voila! Based on his sales course exam score, Rick Thomas can expect to bring in about $12,000 in revenue in the next three months. With more people analyzed (we only have 10 in this example), the greater confidence one can have in the correlation coefficients obtained and the predictive equations garnered. In “real life” I would want as many points of data as possible: hundreds of salesperson data points or more.

I will focus on content validity in my next, so stay tuned!

Twitter: A Job Analysis Tool?


Posted by Greg Pope

I was talking recently with Sean Farrell, a manager of Evaluation and Assessment at a global professional services firm. Sean mentioned an interesting idea that I’d like to share with you, and I’d like to know what you think of it.

Sean recently signed up for a Twitter account. Observing how easy it is for people to post updates and comments there, he began to wonder about how an industrial psychologist could use Twitter. He found a Twitter application to use on his Blackberry, began to search through the options, and came across a function that would remind him to update his tweets on a timed schedule, say every 30 or 60 minutes. Then it hit him! Sean thought perhaps Twitter could be a very useful tool for collecting job task information. This idea made sense to me! I wanted to hear more about what Sean was thinking.

Job analysis is an important part of building valid assessments but in practice it is very difficult to capture good job analysis information. One technique cited in text books is to have job incumbents complete a work journal that captures what they are doing at various times of the work day. Often this technique is viewed as too time consuming and cumbersome for employees to complete. Sean thought: what if we were to ask employees to tweet every 15 or 30 minutes and explain what they are doing at that moment? The person conducting the study could ‘follow’ all the employees and have an instant combined view of tasks completed throughout the day.

twitter-logoIf today’s emerging workforce is already familiar with Twitter and finds it a fun activity then perhaps employees would not mind participating in a Twitter-based job analysis. I think this potential application of Twitter that Sean came up with is really interesting and could be a great way to augment traditional job task analysis information collection via surveys and other means. I want to throw it out there for discussion. Does anyone else think this approach could have merit and want to try it?