How many questions should you have in a web survey?
Web surveys offer a quick, effective means of gathering data and attitudes that can help you make decisions and improvements. But how many questions should you ask? What is the best length for a web survey? Here are some tips:
The best survey length depends on the survey purpose and audience, but here are some useful research findings:
- The market research industry has studied ideal survey length in detail. In such surveys participants are often panel members or people with time who can be motivated or incentivized to answer longish surveys. A debated but often quoted rule of thumb in market research is that 20 minutes is about as long as a typical person can concentrate on a survey and so surveys should be no longer than 20 minutes.
- In typical web surveys, dropout rates increase with a larger number of questions. For example one controlled study found a drop-out rate of 29 percent on a 42-question web survey compared to a smaller dropout rate 23 percent on a 20-question one.
- In long web surveys, participants often reduce time spent answering later questions, which can mean less accurate answers. This is an example of satisficing – participants not thinking too hard about how to answer but just giving an answer. Survey Monkey did an analysis of 100,000 real-world web surveys and found that for surveys of 3 – 10 questions, participants spent an average of 30 seconds answering each question, whereas for surveys of 26 – 30 questions, participants spent an average of 19 seconds. So a longer survey may get lower-quality answers.
- Task difficulty also matters. Shorter isn’t always better. Research (for example here) identifies that difficulty matters as well as length. Participants may abandon a survey when faced with too hard questions, when they would be willing to fill in a longer, less challenging survey.
- Mobile users often have a reduced attention span, and it can take longer to answer questions on a smartphone than on a PC. One experienced commentator suggests that surveys take 20 – 30 percent longer on a mobile device.
So how long should your survey be?
There is no single right answer to this question, here are some tips:
1. A key factor is the engagement of your participants. You can risk a longer survey if your participants are motivated. For example participants who have just undergone a three day course will be more motivated to fill in a longer survey about it than someone who’s just done a short e-learning session.
2. Consider using branching to skip any unneeded questions.
3. Ask concise questions without lengthy explanations, this will reduce the apparent length of the survey.
4. Pretest your survey to try to remove difficult or confusing questions – a longer, clearer survey is better than a shorter, confusing one.
5. If your survey covers very different topics, consider breaking it down into two or more shorter surveys.
6. Make sure results for each question are actionable. There is no point asking questions where you aren’t going to take action depending on what you discover. Participants may disengage if their answers don’t seem likely to be useful .
7. Look at each question and check you really need it. As your survey length increases, your response rate will drop and the quality of the answers may reduce. Work out for each question, whether you need the data badly enough to live with the drop in quality. Ask as few questions as you need – some successful surveys (e.g. Net Promoter Score ) just ask one question. Very often an effective and actionable survey can be ten questions or less.
Want to learn more about survey techniques? I will be presenting a session on harnessing the power of your surveys at the 2016 Questionmark Conference in Miami April 12-15. There’s only 1 week left to take advantage of our early-bird discount. Sign up before January 21 and save $200! I look forward to seeing you there!