New white paper: Assessment Results You Can Trust

John Kleeman HeadshotPosted by John Kleeman

Questionmark published an important white paper about why trustable assessment results matter and about how an assessment management system like Questionmark’s can help you make your assessments valid and reliable — and therefore trustable.

The white paper, which I wrote together with Questionmark CEO Eric Shepherd, explains that trustable assessment results must be both valid (measuring what you are looking for them to measure) and reliable (consistently measuring what you want to be measured).

The paper draws upon the metaphor of a doctor using results from a blood test to diagnose an illness and then prescribe a remedy. Delays will occur if the doctor orders the wrong test, and serious consequences could result if the test’s results are untrustworthy. Using this metaphor, it is easy to understand the personnel and organizational risks that can stem from making decisions based on untrustworthy results. If you assesses someone’s knowledge, skill or competence for health and safety or regThe 6 stages of trustable results; Planning assessment, Authoring items, Assembling assessment, Pilot and review, Delivery, Analyze resultsulatory compliance purposes, you need to ensure that your assessment instrument is designed correctly and runs consistently.

Engaging subject matter experts to generate questions to measure the knowledge, skills and abilities required to perform essential tasks of the job is essential in creating the initial pool of questions. However, subject matter experts are not necessarily experts in writing good questions, so an effective authoring system requires a quality control process which allows assessment experts (e.g. instructional designers or psychometricians) to easily review and amend assessment items.

For assessments to be valid and reliable, it’s necessary to follow structured processes at each step from planning through authoring to delivery and reporting.

The white paper covers these six stages of the assessment process:

  • Planning assessment
  • Authoring items
  • Assembling assessment
  • Pilot and review
  • Delivery
  • Analyze results

Following the advice in the white paper and using the capabilities it describes will help you produce assessments that are more valid and reliable — and hence more trustable.
Modern organizations need their people to be competent.

Would you be comfortable in a high-rise building designed by an unqualified architect? Would you fly in a plane whose pilot hadn’t passed a flying test? Would you let someone operate a machine in your factory if they didn’t know what to do if something went wrong? Would you send a sales person out on a call  if they didn’t know what your products do? Can you demonstrate to a regulatory authority that your staff are competent and fit for their jobs if you do not have trustable assessments?

In all these cases and many more, it’s essential to have a reliable and valid test of competence. If you do not ensure that your workforce is qualified and competent, then you should not be surprised if your employees have accidents, cause your organization to be fined for regulatory infractions, give poor customer service or can’t repair systems effectively.

To download the white paper, click here.

John will be talking more about trustable assessments at our 2015 Users Conference in Napa next month. Register today for the full conference, but if you cannot make it, make sure to catch the live webcast.

Learning Styles: Fiction?

Doug Peterson HeadshotPosted By Doug Peterson

Last week, I wrote about learning styles and the importance many educators place on them. Today, let’s look at the downside of this approach.

Do a Google search on “debunking 4 learning styles” and you’ll find a lot of information. For example, a few years ago the Association for Psychological Science published an article stating that there is no scientific support for learning styles. But there are a couple of points in this article that I would like to bring out.

The first is that the article isn’t really saying that the learning styles theory has been disproved: it’s saying the theory hasn’t been correctly proven. In other words, learning styles may still exist, but the proponents of the theory simply haven’t proven it yet. That’s different from “proven not to exist at all.”

Second, note the little bit that says “the participants would need to take the same test at the end of the experiment.” We know that for an assessment to be fair, valid and reliable, one of the things it must do is allow the participant to display his/her level of knowledge, skill or ability without interference and without testing multiple skills simultaneously (like reading comprehension along with the actual knowledge objective).

So how should we be looking at the relationship between learning styles and assessments? Should proponents of learning styles need to use assessments that take them into consideration? If a person is a visual learner would they be better able to communicate their understanding with a visual question—say a Hot Spot—than with a multiple choice question? And maybe an auditory learner would better communicate his/her understanding with a spoken answer. Would forcing a visual learner to prove their understanding in a non-visual way be fair? Would it truly be testing not only their knowledge? Or would it also be testing their ability to overcome the learning style barrier presented by the question itself?

Those who don’t support the learning style theory feel that anyone can learn from any presentation style—people just have preferred styles. In other words, they feel that the evidence shows that if you had two groups who identify as visual learners, and they both learned the same subject matter but one group learned it visually while the subject matter was presented differently to the second group, both groups would still end up learning the same amount. Their learning style is not a limitation (so much so that they can’t learn as much or as well when the material is presented in other styles), it’s just a preference.

I can’t say that I accept learning styles as fact, but I also can’t say that I believe they are fiction. What I can say is that I believe that learning has to do with two things:

1.       Engagement
2.       Learner motivation

I don’t believe that “learning styles” and “engagement” are the same thing. I can see where, assuming that learning styles exist, it would be easier to engage a visual learner with visual content, but if you have boring visual content, even a visual learner will not learn. I also believe that a podcast done really well can engage a (supposedly) visual or tactile learner. True, according to the theory, the visual or tactile learner may not learn as much as when the material is presented in their style, but I think you get my point that learning must be engaging, and that engagement is independent of learning style.

My experience has also shown me that when a learner is motivated, nothing will stand in his or her way. If passing that eLearning course means a promotion and a raise, that auditory learner will do what it takes to learn the material and pass, even if the material is nothing but charts and graphs. Conversely, if the visual learner couldn’t care less about the material, the greatest graphs in the world won’t make one whit of difference.

I would love to hear your thoughts and opinions on learning styles. Do you think they’re real, and that a learner simply cannot learn as well from material not presented in their style as they can from material that is?? Or do you think that learning style is more of a preference, and that learning will take place regardless of the way in which it is presented as long as it is engaging and the learner is motivated?