How many errors can you spot in this survey question?
Posted by John Kleeman
Tests and surveys are very different. In a test, you look to measure participant knowledge or skill; you know what answer you are looking for, and generally participants are motivated to answer well. In a survey, you look to measure participant attitude or recollection; you don’t know what answer you are looking for, and participants may be disinterested.
Writing good surveys is an important skill. If you’re interested in how to write good surveys of opinion and attitude in training, learning, compliance, certification, based on research evidence, you might be interested in a webinar I gave titled, “Designing Effective Surveys.” Click HERE for the webinar recording and slides.
In the meantime, here’s a sample survey question. How many errors can you spot in the question?
There are quite a few errors. Try to count the errors before you look at my explanation below!!
I count seven errors:
- I am sure you got the mis-spelling of “quality”. If you mis-spell something in a survey question, it indicates to the participant that you haven’t taken time and trouble writing your survey, so there is little incentive for them to spend time and trouble answering.
- It’s not usually sensible to use the word “always” in a survey question. Some participants make take the statement literally, and it’s much more likely that webinars are usually excellent than that every single one is excellent.
- The question is double-barreled. It’s asking about material AND presentation quality. They might be different. This really should be two questions to get a consistent answer.
- The “Agree” in “Strongly Agree” is capitalized but not in other places, e.g. “Slightly agree”. Capitalization should be equal in every part of the scale.
You can see these four errors highlighted below.
Is that all the errors? I count three more, making a total of seven:
- The scale should be balanced. Why is there a “Slightly agree” and not a “Slightly disagree”?
- This is a leading or “loaded” question, not a neutral one, it encourages you to a positive answer. If you genuinely want to get people’s opinion in a survey question, you need to ask it without encouraging the participant to answer a particular way.
- Lastly, any agree/disagree question has acquiescence bias. Research evidence suggests that some participants are more likely to agree when answering survey questions. Particularly those who are more junior or less educated who may tend to think that what is asked of them might be true. It would be better to word this question to ask people to rate the webinars rather than agree with a statement about them.
Did you get all of these? I hope you enjoyed this little exercise. If you did, I explain more about this and good survey practice in our Designing Effective Surveys webinar, click HERE for the webinar recording and slides.