How many questions should you have in a web survey?

John Kleeman Headshot

Posted by John Kleeman

Web surveys offer a quick, effective means of gathering data and attitudes that can help you make decisions and improvements. But how many questions should you ask? What is the best length for a web survey? Here are some tips:

Want to learn more about survey techniques? I will be presenting a session on harnessing the power of your surveys at the 2016 Questionmark Conference in Miami April 12-15.

Research evidence

The best survey length depends on the survey purpose and audience, but here are some useful research findings:

  • The market research industry has studied ideal survey length in detail. In such surveys participants are often panel members or people with time who can be motivated or incentivized to answer longish surveys. A debated but often quoted rule of thumb in market research is that 20 minutes is about as long as a typical person can concentrate on a survey and so surveys should be no longer than 20 minutes.
  • In typical web surveys, dropout rates increase with a larger number of questions. For example one controlled study found  a drop-out rate of 29 percent on a 42-question web survey compared to a smaller dropout rate 23 percent on a 20-question one.
  • In long web surveys, participants often reduce time spent answering later questions, which can mean less accurate answers. This is an example of satisficing – participants not thinking too hard about how to answer but just giving an answer. Survey Monkey did an analysis of 100,000 real-world web surveys and found that for surveys of 3 – 10 questions, participants spent an average of 30 seconds answering each question, whereas for surveys of 26 – 30 questions, participants spent an average of 19 seconds.  So a longer survey may get lower-quality answers.
  • Task difficulty also matters. Shorter isn’t always better. Research (for example here) identifies that difficulty matters as well as length. Participants may abandon a survey when faced with too hard questions, when they would be willing to fill in a longer, less challenging survey.
  • Mobile users often have a reduced attention span, and it can take longer to answer questions on a smartphone than on a PC. One experienced commentator suggests that surveys take 20 – 30 percent longer on a mobile device.

So how long should your survey be?

There is no single right answer to this question, here are some tips:

Editing a jump block - choosing to skip to end of assessment if previous question was not applicable1. A key factor is the engagement of your participants. You can risk a longer survey if your participants are motivated. For example participants who have just undergone a three day course will be more motivated to fill in a longer survey about it than someone who’s just done a short e-learning session.

2. Consider using  branching to skip any unneeded questions.

3. Ask concise questions without lengthy explanations, this will reduce the apparent length of the survey.

4. Pretest your survey to try to remove difficult or confusing questions – a longer, clearer survey is better than a shorter, confusing one.

5. If your survey covers very different topics, consider breaking it down into two or more shorter surveys.

6. Make sure results for each question are actionable. There is no point asking questions where you aren’t going to take action depending on what you discover. Participants may disengage if their answers don’t seem likely to be useful .

7. Look at each question and check you really need it. As your survey length increases, your response rate will drop and the quality of the answers may reduce.  Work out for each question, whether you need the data badly enough to live with the drop in quality. Ask as few questions as you need – some successful surveys (e.g. Net Promoter Score ) just ask one question. Very often an effective and actionable survey can be ten questions or less.

Want to learn more about survey techniques? I will be presenting a session on harnessing the power of your surveys at the 2016 Questionmark Conference in Miami April 12-15. There’s only 1 week left to take advantage of our early-bird discount. Sign up before January 21 and save $200! I look forward to seeing you there!

Read more >

Certification in the Cloud and the Move to Online Proctoring: An interview with SAP’s manager of global certification

John Kleeman Headshot

Posted by John Kleeman

I recently interviewed Ralf Kirchgaessner, SAP’s manager of global certification, about how the cloud is changing SAP certification. This is a shortened version of my conversation with Ralf. To read the full previously published post, check out this SAP blog.

John: What are the key reasons why SAP has a certification program?

Ralf: The overall mission of the program is that every SAP solution should be implemented and supported ideally by a certified SAP resource. This is to ensure that implementation projects go well for customers, and to increase customer productivity while reducing their operating costs. Customers value certification. In a survey of SAP User Group customers in Germany and the US, 80 percent responded that it was very important to have their employees certified and over 60 percent responded that certification was one of the criteria used to select external consultants for implementation projects.

John: What important trends do you see in high tech and IT certification?

Ralf: What comes first to the mind is the move to the cloud. Throughout the technology industry, the cloud drives flexibility and making everything available on demand. One aspect of this is that release cycles are getting quicker and quicker.

For certification, this means that consultants and others have to show that they are always up to date and are certified on the latest release. It’s not enough to become certified once in your lifetime: you have to continually learn and stay up to date. But of course if you are taking certification exams more often, certification costs have to be much lower. In some regions, people have to travel large distances to get to a test centre. With more frequent certification, it’s not practical to travel to a testing centre every time you take a certification. So our aim is to allow certification anytime and anywhere using the cloud.

John: How does online proctoring work for the candidate?

Ralf: A remote proctor monitors the candidate via a webcam, and there are a lot of security checks done by the proctor and by the system. For example, a secure browser is used, the candidate has to do a 360 degree check of his or her room, and there are lots of specific controls. For instance, you aren’t allowed to read the questions silently with your lips in case someone is watching or listening.

The great advantage to the candidate is flexibility. If someone says, “I’d like to do my exam in the middle of the night or on weekends because during the week I’m so busy with my project,” they can. They might say that they’d like to do their exam on Saturday afternoon: “After spending two hours playing with my kids, I’m relaxed to do my exam!” It’s such a flexible way to get certified and to quickly demonstrate that they have up-to-date knowledge and are allowed to provision customer systems.

John: Who benefits from certification in the cloud? Candidates, customers, partners or SAP?

Ralf: Of course, I think all benefit! Candidates have flexibility and lower cost. Customers can be sure that partner consultants who work for them are enabled and up to date. For partners, it’s a competitive advantage to show that their consultants are up to date, especially for new technologies like S/4HANA and Simple Finance. A partner is much more likely to be chosen to deploy new technologies if they can demonstrate that they have several consultants already certified in something that’s just been released. And for SAP, our goal is to have engaged consultants, happy partners and lower support costs. So everyone genuinely benefits.

John: What are some of the challenges?

Ralf: One example is that it’s important in cloud certification to get data protection right. SAP have very detailed requirements that we ensure our vendors like Questionmark meet.

Security is also a challenge. You need to prevent cheating and stealing questions.  And interfaces and integration need to be right. We have worked out how we get the data from our HR systems, how people book and subscribe to exams and then how they can authenticate with single sign-on into the certification hub to take cloud exams.

The delta concept also gives challenges. You need very precise pre-requisite management logic, where the certification software checks for example that, if you want to take the delta exam, you have already passed the core exam. It also can sometimes be difficult to prepare a good delta exam, particularly if a new release has very specific or detailed features, including some that apply in only some industries.

Lastly, providing seamless support is a challenge when using multiple vendors. The candidate doesn’t care where a problem happened: he or she just wants it fixed.

John: Where do you see the long term future of high-tech certification? Will there still be test centres, or will all certification be done via the cloud?

Ralf: Test centres won’t disappear at once, but there is a trend of moving from classroom-based learning and testing to learning and certification in the cloud. The future will belong to anytime, anywhere testing. The trend is for test centre use to decline, but it won’t happen overnight!

John: If another organization is thinking of moving towards certification in the cloud, what advice would you give them?

Ralf: Ensure that you are aware of the challenges I mentioned and can deal with them. And do some pilots before you try to scale.

Interested in learning more about Online Proctoring? I will be presenting a session on ensuring exam integrity with online proctoring at Questionmark Conference 2016: Shaping the Future of Assessment in Miami, April 12-15. I look forward to seeing you there! Click here to register and learn more about this important learning event.

Questionmark customers still safe independent of Safe Harbor

eu flagJohn Kleeman HeadshotPosted by John Kleeman

Since my earlier post, Is Safe Harbor still safe for assessment data?, the European Court of Justice has ruled that the Safe Harbor mechanism under which many transfers of personal data from Europe to the US take place is no longer valid. Here is how Questionmark customers typically remain safe in spite of this invalidation.

What is the EU-US Safe Harbor Framework?

The EU-US Safe Harbor Framework was established by the European Commission and the US government in 2000 to facilitate transfers of personal data from the EU to eligible US companies that certify to and comply with the Safe Harbor principles. You can see more about Safe Harbor at the US government website: http://www.export.gov/safeharbor/.

What did the European Court of Justice decide on 6 October 2015 regarding the EU-US Safe Harbor Framework?

Essentially, the European Court of Justice decision means that the EU-US Safe Harbor Framework does not provide a valid legal basis within the European Union for transfers of personal data from Europe to the US. The Court reached this conclusion by invalidating the European Commission’s 2000 decision approving Safe Harbor as adequately protecting personal data.

What does the European Court of Justice decision mean for the use of Questionmark OnDemand by organizations based in the EU?

Questionmark has been following these developments and has been aware of concerns about Safe Harbor for some time. Questionmark has measures in place with its non-EU subcontractors who hold OnDemand data. These arrangements include the EU Model Clauses which were not invalidated by the European Court of Justice.

If you are using our European OnDemand service, then all data is hosted in the European Union. In the rare cases that data leaves the European Union, for example for troubleshooting purposes, we have EU Model Clauses in place with any non-EU subcontractors to ensure that any such data transfer is legal, and we regularly review the security of such subcontractors.

Most EU customers of Questionmark use our European OnDemand service, but if you are an EU customer using our US OnDemand service, then this service is delivered from our US data center. However, providing your contract with or invoice from Questionmark is with Questionmark Computing Limited, the UK headquarters company of Questionmark, then you should have no cause for concern. Questionmark is legally obliged to follow UK data protection law. Also, we have EU Model Clauses in place with Questionmark Corporation, and through the corporation with the US data center that delivers the US OnDemand service. So we do not rely on Safe Harbor for personal data stored within Questionmark OnDemand.

What does the European Court of Justice decision mean for the use of Questionmark OnDemand by an organization based outside of the EU?

Organizations without EU personal data  will not be concerned about this ruling, which only applies to transfers of personal data from the EU. Questionmark continues to place the highest value on security for all our customers, and this legal ruling doesn’t change that.

If you have EU personal data and you are not based in the EU, please raise any questions you may have about this with your account manager at Questionmark. We will do everything we can to help you.

What about the US Patriot Act? Is my data stored with Questionmark vulnerable to legal action under the Patriot Act?

Unlike many technology vendors, Questionmark is headquartered in Europe. This means that the services we offer from Europe to our European customers are resistant to legal action within the US, such as under the Patriot Act.

Questionmark’s European OnDemand Service is run by a UK company using a European owned data center operator.

What if I am using Questionmark Perception?

If you are using Questionmark Perception your organization hosts the data and is responsible for compliance with local, and potentially, international laws. So so you need to seek independent legal advice as to whether your systems are configured correctly and whether your subcontractors have signed up to the EU model clauses. You will not normally need to send personal data to Questionmark, however, it may be necessary for us to ask for a copy of your Perception database to troubleshoot an issue, and if you do so, we will treat this securely. If you have any concerns about this process as a result of the Safe Harbor ruling, please raise with your account manager. You may also want to consider migrating to Questionmark OnDemand – please contact your account manager for further information.

This blog post has been written and is provided for general informational purposes only. The content of this blog does not constitute legal advice of a general or specific nature, and readers should consult an attorney to establish how these recent developments impact their organizations.

Unlocking website security

Steve Lay HeadshotPosted by Steve Lay

As a product manager at Questionmark, one of the questions that I’m increasingly being asked is about support for specific versions of SSL and TLS. These abbreviations refer to different flavours of the ‘https’ protocol that keeps your web browsing secure. Questionmark’s OnDemand service no longer supports the older SSL protocol. To understand why, read on…

In this post I’ll focus on the privacy aspect of secure websites only —the extent to which communication is protected from eavesdroppers. Issues of trust are just as important, but I’ll have to discuss those in a future post.

Most browsers display a padlock icon by the web address or the site name to indicate that communication between your browser and the server is encrypted for privacy. Just as with real padlocks, though, there are stronger and weaker forms of encryption. The difference is too subtle for most browsers to show. In practice, browsers adopt a strategy of attempting to use the strongest type of encryption protocol they can, falling back to weaker methods if required. In Internet Explorer you can even configure these settings under the Advanced tab of your internet options:

qm comp 1As you can see, there are five different encryption protocols listed, in increasing order of strength. Generally speaking, TLS is better than SSL and more recent versions of TLS are better still. Published attacks on these protocols typically enable someone who can view network traffic to decrypt some or even all of the information passing over the ‘secure connection’. This type of scenario is called a ‘man in the middle attack’ because the eavesdropper stands in between your browser and the website it is communicating with.

If your browser always chooses the best encryption available, why would you want to configure the specific protocols it supports? Unfortunately, the very first part of the communication between your browser and the website is more vulnerable. The two systems have to agree on an encryption protocol to use before they can be truly private. In some special cases it is possible for a man in the middle to intervene and force a weaker protocol to be negotiated. By configuring your browser to support only stronger protocols, you can ensure that your browser is never tricked this way.

Here at Questionmark, we care about your security too! If a protocol like SSLv3 is considered vulnerable to interception, shouldn’t the server refuse to use it as well? Yes, it should. In fact, we don’t support SSL versions 2 and 3 for this very reason.

For this blog post I’ve focused on the most visible aspect of the security protocol. In practice, there lots of subtle differences in the way each protocol can be configured. If you use Google’s Chrome browser you can click on the padlock to reveal information about connection security.

qm compNotice that this connection uses TLS 1.2, but there is even more detail reported concerning the specific cryptographic algorithms used. Sites like www.ssllabs.com have almost 50 separate check points that they can report on for a public-facing secure website! Staying on top of all this configuration complexity is critical to keeping websites secure.

Unfortunately, sometimes we have to strengthen security in such a way that compatibility with older browsers is sacrificed. For example, according to the latest simulation results, Internet Explorer version 6 (running on Windows XP) is no longer able to successfully negotiate a secure connection with our OnDemand service.

In practice, an overwhelming majority of users use more modern browsers (or have access to one), so the web remains both secure and usable. Perhaps a greater cause of concern is older applications that are integrated with our APIs. It is just as important to keep these applications up to date. For example, applications that use older versions of Java, such as Java 6 or have their Java runtime configuration options set inappropriately might have problems communicating to the same high standards. If you are running a custom integration and are concerned about future compatibility, please get in touch.

This is a developing field. New ways of exploiting older protocols and cryptographic algorithms are being found by security researchers all the time, and the bad guys aren’t far behind. Our security specialists at Questionmark constantly monitor best practice and update the configuration of our OnDemand service to keep your communications safe.

Is There Value in Reporting Subscores?

Austin Fossey-42Posted by Austin Fossey

The decision to report subscores (reported as Topic Scores in Questionmark’s software) can be a difficult one, and test developers often need to respond to demands from stakeholders who want to bleed as much information out of an instrument as they can. High-stakes test development is lengthy and costly, and the instruments themselves consume and collect a lot of data that can be valuable for instruction or business decisions. It makes sense that stakeholders want to get as much mileage as they can out of the instrument.

It can be anticlimactic when all of the development work results in just one score or a simple pass/fail decision. But that is after all what many instruments are designed to do. Many assessment models assume unidimensionality, so a single score or classification representing the participant’s ability is absolutely appropriate. Nevertheless, organizations often find themselves in the position of trying to wring out more information. What are my participants’ strengths and weaknesses? How effective were my instructors? There are many ways in which people will try to repurpose an assessment.

The question of whether or not to report subscores certainly falls under this category. Test blueprints often organize the instrument around content areas (e.g., Topics), and these lend themselves well to calculating subscores for each of the content areas. From a test user perspective, these scores are easy to interpret, and they are considered valuable because they show content areas where participants perform well or poorly, and because it is believed that this information can help inform instruction.

But how useful are these subscores? In their article, A Simple Equation to Predict a Subscore’s Value, Richard Feinberg and Howard Wainer explain that there are two criteria that must be met to justify reporting a subscore:

  • The subscore must be reliable.
  • The subscore must contain information that is sufficiently different from the information that is contained by the assessment’s total score.

If a subscore (or any score) is not reliable, there is no value in reporting it. The subscore will lack precision, and any decisions made on an unreliable score might not be valid. There is also little value if the subscore does not provide any new information. If the subscores are effectively redundant to the total score, then there is no need to report them. The flip side of the problem is that if subscores do not correlate with the total score, then the assessment may not be unidimensional, and then it may not make sense to report the total score. These are the problems that test developers wrestle with when they lie awake at night.

Excerpt from Questionmark’s Test Analysis Report showing low reliability of three topic scores.

As you might have guessed from the title of their article, Feinberg and Wainer have proposed a simple, empirically-based equation for determining whether or not a subscore should be reported. The equation yields a value that Sandip Sinharay and Shelby Haberman called the Value Added Ratio (VAR). If a subscore on an assessment has a VAR value greater than one, then they suggest that this justifies reporting it. All of the VAR values that are less than one, should not be reported. I encourage interested readers to check out Feinberg and Wainer’s article (which is less than two pages, so you can handle it) for the formula and step-by-step instructions for its application.

 

South African Users Conference Programme Takes Shape

Chloe MendoncaPosted by Chloe Mendonca

In just five weeks, Questionmark users and other learning professionals will gather in Midrand for the first South African Questionmark Users Conference.

Delegates will enjoy a full programme, from case studies to features and functions sessions on the effective use of Questionmark technologies.

There will also be time for networking during lunches and our Thursday evening event.

Here are some of the sessions you can look forward to:

  • Case Study: Coming alive with Questionmark Live: A mind shift for lecturers – University of Pretoria
  • Case Study: Lessons and Discoveries over a decade with Questionmark – Nedbank
  • Case Study: Stretching the Boundaries: Using Questionmark in a High-Volume Assessment Environment – University of Pretoria
  • Features and Functions: New browser based tools for collaborative authoring – Overview and Demonstrations
  • Features and Functions: Analysing and Sharing Results with Stakeholders: Overview of Key New Questionmark Reporting and Analytics features
  • Features and Functions: Extending the Questionmark Platform: Updates and overviews of APIs, Standards Support, and integrations with third party applications
  • Customer Panel Discussion: New Horizons for eAssessment

You can register for the conference online or visit our website for more information.

We look forward to seeing you in August!

Chloe-Banner-SA-Conference-2014