10 Reasons Why Frequent Testing Makes Sense

Posted by John Kleeman

It matters to society, organizations and individuals that test results are trustable. Tests and exams are used to make important decisions about people and each failure of test security reduces that trustworthiness.

There are several risks to test security, but two important ones are identity fraud and getting help from others. With identity fraud, someone asks a friend to take the test for them or pays a professional cheater to take the test and pretend to be them. With getting help from others, a test-taker subverts the process and gets a friend or expert to help them with the test, feeding them the right answers. In both cases, this makes the individual test result meaningless and detracts from the value and trustworthiness of the whole assessment process.

There are lots of mitigations to these risks – checking identity carefully, having well trained proctors, using forensics or other reports and using technical solutions like secure browsers – and these are very helpful. But testing more frequently can also reduce the risk: let me explain.

Suppose you just need to pass a single exam to get an important career step – certification, qualification or other important job requirement, then the incentive to cheat on that one test is large. But if you have a series of smaller tests over a period, then it’s more hassle for a test taker to conduct identity fraud or to get help from others each time. He or she would have to pay the proxy test taker several times.  And make sure the same person is available in case photos are captured. And for the expert help you also must reach out more often, and evade whatever security there is each time

There are other benefits too; here is a list of ten reasons why more frequent testing makes sense:

  1. More reliable. More frequent testing contributes to more reliable testing. A single large test is vulnerable to measurement error if a test taker is sick or has an off day, whereas that is less likely to impact frequent tests.
  2. More up to date. With technology and society changing rapidly, more frequent tests can make tests more current. For instance, some IT certification providers create “delta” tests measuring understanding of their latest releases and encourage people to take quarterly tests to ensure they remain up to date.
  3. Less test anxiety. Test anxiety can be a big challenge to some test takers (see Ten tips on reducing test anxiety for online test-takers), and more frequent tests means less is at stake for each one, and so may help test takers be less anxious.
  4. More feedback. More frequent tests give feedback to test takers on how well they are performing and allow them to identify training or continuing education to improve.
  5. More data for testing organization. In today’s world of business intelligence and analytics, there is potential for correlations and other valuable insight from the data of people’s performance in a series of tests over time.
  6. Encourages test takers to target retention of learning. We all know of people who cram for an exam and then forget it afterwards. More frequent tests encourage people to plan to learn for the longer term.
  7. Encourages spaced out learning. There is strong evidence that learning at spaced out intervals makes it more likely knowledge and skills will be retained. Periodic tests encourage revision at regular intervals and so make it more likely that learning will be remembered.
  8. Testing effect. There is also evidence that tests themselves give retrieval practice and aid retention and more frequent tests will give more such practice.
  9. More practical. With online assessment software and online proctoring, it’s very practical to test frequently, and no longer necessary to bring test takers to a central testing center for one off large tests.
  10. Harder to cheat. Finally, as described above, more frequent testing makes it harder to use identity fraud or to get help from others, which reduce cheating.

I think we’re seeing a slow paradigm shift from larger testing events that happen at a single point in time to smaller, online testing events happening periodically. What do you think?

xAPI: A Way to Enable Learning Analytics

Posted by John Kleeman

Many organizations train and test individuals to ensure they have the right skills and competencies. In doing so, they amass vast amounts of data, which can be used to identify further training opportunities and improve performance. One way of managing this data is to use the Experience API (or xAPI) to pass data from disparate systems into a central Learning Record Store.

xAPI is maintained by the United States Advanced Distributed Learning Initiative (see www.adlnet.gov) and many Questionmark users have requested that we support xAPI so that they can export test data for analysis. For this reason, we’re pleased to let you know that earlier this year, we released our xAPI Connector for OnPremise and OnDemand customers. The integration lets the Questionmark platform connect and ‘talk’ to Learning Record Stores, creating an agile and effective learning and development ecosystem.

The challenges organizations face

For any organization, measuring the competence of employees or consultants through assessment is an essential element of ensuring the team is capable and fit-for-purpose. During this process, organizations collect large amounts of data that needs to be stored under strict data privacy regulations.

Once employers have control of learning and assessment data, it can then be interrogated to analyze employees and the effectiveness of training programs. With Questionmark’s xAPI integration, customers will now be able to transfer data from the assessment platform to their Learning Record Store.

What xAPI does

xAPI provides a standard means for collecting data from training and assessment experiences. The specification allows different systems to communicate and share data, which can then be stored and analyzed. This helps organizations to make better decisions by collecting, tracking, and quantifying learning activities to see what works and what doesn’t.

Organizations are increasingly investing in Learning Record Stores to host and analyze learning and assessment data. With xAPI, Questionmark customers will now to be able to send assessment data directly to their Learning Record Stores, so that they can measure the impact of learning and development activities and maximize the impact of their investment.

xAPI offers universal integration, meaning users can store data anywhere. Reporting across multiple geographies is easy, so users can analyze, compare and contrast data. The data is also presented in a universal format, making it easy to understand and interpret. This provides a solid starting point for big data learning analytics. And, as an assessment technology provider, Questionmark has widened its footprint in the total learning ecology by releasing the xAPI functionality.

If you’d like to find out more about the full range of assessment features that Questionmark offers, contact us or request a demo.

How many questions should you have in a web survey?

John Kleeman Headshot

Posted by John Kleeman

Web surveys offer a quick, effective means of gathering data and attitudes that can help you make decisions and improvements. But how many questions should you ask? What is the best length for a web survey? Here are some tips:

Want to learn more about survey techniques? I will be presenting a session on harnessing the power of your surveys at the 2016 Questionmark Conference in Miami April 12-15.

Research evidence

The best survey length depends on the survey purpose and audience, but here are some useful research findings:

  • The market research industry has studied ideal survey length in detail. In such surveys participants are often panel members or people with time who can be motivated or incentivized to answer longish surveys. A debated but often quoted rule of thumb in market research is that 20 minutes is about as long as a typical person can concentrate on a survey and so surveys should be no longer than 20 minutes.
  • In typical web surveys, dropout rates increase with a larger number of questions. For example one controlled study found  a drop-out rate of 29 percent on a 42-question web survey compared to a smaller dropout rate 23 percent on a 20-question one.
  • In long web surveys, participants often reduce time spent answering later questions, which can mean less accurate answers. This is an example of satisficing – participants not thinking too hard about how to answer but just giving an answer. Survey Monkey did an analysis of 100,000 real-world web surveys and found that for surveys of 3 – 10 questions, participants spent an average of 30 seconds answering each question, whereas for surveys of 26 – 30 questions, participants spent an average of 19 seconds.  So a longer survey may get lower-quality answers.
  • Task difficulty also matters. Shorter isn’t always better. Research (for example here) identifies that difficulty matters as well as length. Participants may abandon a survey when faced with too hard questions, when they would be willing to fill in a longer, less challenging survey.
  • Mobile users often have a reduced attention span, and it can take longer to answer questions on a smartphone than on a PC. One experienced commentator suggests that surveys take 20 – 30 percent longer on a mobile device.

So how long should your survey be?

There is no single right answer to this question, here are some tips:

Editing a jump block - choosing to skip to end of assessment if previous question was not applicable1. A key factor is the engagement of your participants. You can risk a longer survey if your participants are motivated. For example participants who have just undergone a three day course will be more motivated to fill in a longer survey about it than someone who’s just done a short e-learning session.

2. Consider using  branching to skip any unneeded questions.

3. Ask concise questions without lengthy explanations, this will reduce the apparent length of the survey.

4. Pretest your survey to try to remove difficult or confusing questions – a longer, clearer survey is better than a shorter, confusing one.

5. If your survey covers very different topics, consider breaking it down into two or more shorter surveys.

6. Make sure results for each question are actionable. There is no point asking questions where you aren’t going to take action depending on what you discover. Participants may disengage if their answers don’t seem likely to be useful .

7. Look at each question and check you really need it. As your survey length increases, your response rate will drop and the quality of the answers may reduce.  Work out for each question, whether you need the data badly enough to live with the drop in quality. Ask as few questions as you need – some successful surveys (e.g. Net Promoter Score ) just ask one question. Very often an effective and actionable survey can be ten questions or less.

Want to learn more about survey techniques? I will be presenting a session on harnessing the power of your surveys at the 2016 Questionmark Conference in Miami April 12-15. There’s only 1 week left to take advantage of our early-bird discount. Sign up before January 21 and save $200! I look forward to seeing you there!

Read more >

Certification in the Cloud and the Move to Online Proctoring: An interview with SAP’s manager of global certification

John Kleeman Headshot

Posted by John Kleeman

I recently interviewed Ralf Kirchgaessner, SAP’s manager of global certification, about how the cloud is changing SAP certification. This is a shortened version of my conversation with Ralf. To read the full previously published post, check out this SAP blog.

John: What are the key reasons why SAP has a certification program?

Ralf: The overall mission of the program is that every SAP solution should be implemented and supported ideally by a certified SAP resource. This is to ensure that implementation projects go well for customers, and to increase customer productivity while reducing their operating costs. Customers value certification. In a survey of SAP User Group customers in Germany and the US, 80 percent responded that it was very important to have their employees certified and over 60 percent responded that certification was one of the criteria used to select external consultants for implementation projects.

John: What important trends do you see in high tech and IT certification?

Ralf: What comes first to the mind is the move to the cloud. Throughout the technology industry, the cloud drives flexibility and making everything available on demand. One aspect of this is that release cycles are getting quicker and quicker.

For certification, this means that consultants and others have to show that they are always up to date and are certified on the latest release. It’s not enough to become certified once in your lifetime: you have to continually learn and stay up to date. But of course if you are taking certification exams more often, certification costs have to be much lower. In some regions, people have to travel large distances to get to a test centre. With more frequent certification, it’s not practical to travel to a testing centre every time you take a certification. So our aim is to allow certification anytime and anywhere using the cloud.

John: How does online proctoring work for the candidate?

Ralf: A remote proctor monitors the candidate via a webcam, and there are a lot of security checks done by the proctor and by the system. For example, a secure browser is used, the candidate has to do a 360 degree check of his or her room, and there are lots of specific controls. For instance, you aren’t allowed to read the questions silently with your lips in case someone is watching or listening.

The great advantage to the candidate is flexibility. If someone says, “I’d like to do my exam in the middle of the night or on weekends because during the week I’m so busy with my project,” they can. They might say that they’d like to do their exam on Saturday afternoon: “After spending two hours playing with my kids, I’m relaxed to do my exam!” It’s such a flexible way to get certified and to quickly demonstrate that they have up-to-date knowledge and are allowed to provision customer systems.

John: Who benefits from certification in the cloud? Candidates, customers, partners or SAP?

Ralf: Of course, I think all benefit! Candidates have flexibility and lower cost. Customers can be sure that partner consultants who work for them are enabled and up to date. For partners, it’s a competitive advantage to show that their consultants are up to date, especially for new technologies like S/4HANA and Simple Finance. A partner is much more likely to be chosen to deploy new technologies if they can demonstrate that they have several consultants already certified in something that’s just been released. And for SAP, our goal is to have engaged consultants, happy partners and lower support costs. So everyone genuinely benefits.

John: What are some of the challenges?

Ralf: One example is that it’s important in cloud certification to get data protection right. SAP have very detailed requirements that we ensure our vendors like Questionmark meet.

Security is also a challenge. You need to prevent cheating and stealing questions.  And interfaces and integration need to be right. We have worked out how we get the data from our HR systems, how people book and subscribe to exams and then how they can authenticate with single sign-on into the certification hub to take cloud exams.

The delta concept also gives challenges. You need very precise pre-requisite management logic, where the certification software checks for example that, if you want to take the delta exam, you have already passed the core exam. It also can sometimes be difficult to prepare a good delta exam, particularly if a new release has very specific or detailed features, including some that apply in only some industries.

Lastly, providing seamless support is a challenge when using multiple vendors. The candidate doesn’t care where a problem happened: he or she just wants it fixed.

John: Where do you see the long term future of high-tech certification? Will there still be test centres, or will all certification be done via the cloud?

Ralf: Test centres won’t disappear at once, but there is a trend of moving from classroom-based learning and testing to learning and certification in the cloud. The future will belong to anytime, anywhere testing. The trend is for test centre use to decline, but it won’t happen overnight!

John: If another organization is thinking of moving towards certification in the cloud, what advice would you give them?

Ralf: Ensure that you are aware of the challenges I mentioned and can deal with them. And do some pilots before you try to scale.

Interested in learning more about Online Proctoring? I will be presenting a session on ensuring exam integrity with online proctoring at Questionmark Conference 2016: Shaping the Future of Assessment in Miami, April 12-15. I look forward to seeing you there! Click here to register and learn more about this important learning event.

Questionmark customers still safe independent of Safe Harbor

eu flagJohn Kleeman HeadshotPosted by John Kleeman

Since my earlier post, Is Safe Harbor still safe for assessment data?, the European Court of Justice has ruled that the Safe Harbor mechanism under which many transfers of personal data from Europe to the US take place is no longer valid. Here is how Questionmark customers typically remain safe in spite of this invalidation.

What is the EU-US Safe Harbor Framework?

The EU-US Safe Harbor Framework was established by the European Commission and the US government in 2000 to facilitate transfers of personal data from the EU to eligible US companies that certify to and comply with the Safe Harbor principles. You can see more about Safe Harbor at the US government website: http://www.export.gov/safeharbor/.

What did the European Court of Justice decide on 6 October 2015 regarding the EU-US Safe Harbor Framework?

Essentially, the European Court of Justice decision means that the EU-US Safe Harbor Framework does not provide a valid legal basis within the European Union for transfers of personal data from Europe to the US. The Court reached this conclusion by invalidating the European Commission’s 2000 decision approving Safe Harbor as adequately protecting personal data.

What does the European Court of Justice decision mean for the use of Questionmark OnDemand by organizations based in the EU?

Questionmark has been following these developments and has been aware of concerns about Safe Harbor for some time. Questionmark has measures in place with its non-EU subcontractors who hold OnDemand data. These arrangements include the EU Model Clauses which were not invalidated by the European Court of Justice.

If you are using our European OnDemand service, then all data is hosted in the European Union. In the rare cases that data leaves the European Union, for example for troubleshooting purposes, we have EU Model Clauses in place with any non-EU subcontractors to ensure that any such data transfer is legal, and we regularly review the security of such subcontractors.

Most EU customers of Questionmark use our European OnDemand service, but if you are an EU customer using our US OnDemand service, then this service is delivered from our US data center. However, providing your contract with or invoice from Questionmark is with Questionmark Computing Limited, the UK headquarters company of Questionmark, then you should have no cause for concern. Questionmark is legally obliged to follow UK data protection law. Also, we have EU Model Clauses in place with Questionmark Corporation, and through the corporation with the US data center that delivers the US OnDemand service. So we do not rely on Safe Harbor for personal data stored within Questionmark OnDemand.

What does the European Court of Justice decision mean for the use of Questionmark OnDemand by an organization based outside of the EU?

Organizations without EU personal data  will not be concerned about this ruling, which only applies to transfers of personal data from the EU. Questionmark continues to place the highest value on security for all our customers, and this legal ruling doesn’t change that.

If you have EU personal data and you are not based in the EU, please raise any questions you may have about this with your account manager at Questionmark. We will do everything we can to help you.

What about the US Patriot Act? Is my data stored with Questionmark vulnerable to legal action under the Patriot Act?

Unlike many technology vendors, Questionmark is headquartered in Europe. This means that the services we offer from Europe to our European customers are resistant to legal action within the US, such as under the Patriot Act.

Questionmark’s European OnDemand Service is run by a UK company using a European owned data center operator.

What if I am using Questionmark Perception?

If you are using Questionmark Perception your organization hosts the data and is responsible for compliance with local, and potentially, international laws. So so you need to seek independent legal advice as to whether your systems are configured correctly and whether your subcontractors have signed up to the EU model clauses. You will not normally need to send personal data to Questionmark, however, it may be necessary for us to ask for a copy of your Perception database to troubleshoot an issue, and if you do so, we will treat this securely. If you have any concerns about this process as a result of the Safe Harbor ruling, please raise with your account manager. You may also want to consider migrating to Questionmark OnDemand – please contact your account manager for further information.

This blog post has been written and is provided for general informational purposes only. The content of this blog does not constitute legal advice of a general or specific nature, and readers should consult an attorney to establish how these recent developments impact their organizations.

Unlocking website security

Steve Lay HeadshotPosted by Steve Lay

As a product manager at Questionmark, one of the questions that I’m increasingly being asked is about support for specific versions of SSL and TLS. These abbreviations refer to different flavours of the ‘https’ protocol that keeps your web browsing secure. Questionmark’s OnDemand service no longer supports the older SSL protocol. To understand why, read on…

In this post I’ll focus on the privacy aspect of secure websites only —the extent to which communication is protected from eavesdroppers. Issues of trust are just as important, but I’ll have to discuss those in a future post.

Most browsers display a padlock icon by the web address or the site name to indicate that communication between your browser and the server is encrypted for privacy. Just as with real padlocks, though, there are stronger and weaker forms of encryption. The difference is too subtle for most browsers to show. In practice, browsers adopt a strategy of attempting to use the strongest type of encryption protocol they can, falling back to weaker methods if required. In Internet Explorer you can even configure these settings under the Advanced tab of your internet options:

qm comp 1As you can see, there are five different encryption protocols listed, in increasing order of strength. Generally speaking, TLS is better than SSL and more recent versions of TLS are better still. Published attacks on these protocols typically enable someone who can view network traffic to decrypt some or even all of the information passing over the ‘secure connection’. This type of scenario is called a ‘man in the middle attack’ because the eavesdropper stands in between your browser and the website it is communicating with.

If your browser always chooses the best encryption available, why would you want to configure the specific protocols it supports? Unfortunately, the very first part of the communication between your browser and the website is more vulnerable. The two systems have to agree on an encryption protocol to use before they can be truly private. In some special cases it is possible for a man in the middle to intervene and force a weaker protocol to be negotiated. By configuring your browser to support only stronger protocols, you can ensure that your browser is never tricked this way.

Here at Questionmark, we care about your security too! If a protocol like SSLv3 is considered vulnerable to interception, shouldn’t the server refuse to use it as well? Yes, it should. In fact, we don’t support SSL versions 2 and 3 for this very reason.

For this blog post I’ve focused on the most visible aspect of the security protocol. In practice, there lots of subtle differences in the way each protocol can be configured. If you use Google’s Chrome browser you can click on the padlock to reveal information about connection security.

qm compNotice that this connection uses TLS 1.2, but there is even more detail reported concerning the specific cryptographic algorithms used. Sites like www.ssllabs.com have almost 50 separate check points that they can report on for a public-facing secure website! Staying on top of all this configuration complexity is critical to keeping websites secure.

Unfortunately, sometimes we have to strengthen security in such a way that compatibility with older browsers is sacrificed. For example, according to the latest simulation results, Internet Explorer version 6 (running on Windows XP) is no longer able to successfully negotiate a secure connection with our OnDemand service.

In practice, an overwhelming majority of users use more modern browsers (or have access to one), so the web remains both secure and usable. Perhaps a greater cause of concern is older applications that are integrated with our APIs. It is just as important to keep these applications up to date. For example, applications that use older versions of Java, such as Java 6 or have their Java runtime configuration options set inappropriately might have problems communicating to the same high standards. If you are running a custom integration and are concerned about future compatibility, please get in touch.

This is a developing field. New ways of exploiting older protocols and cryptographic algorithms are being found by security researchers all the time, and the bad guys aren’t far behind. Our security specialists at Questionmark constantly monitor best practice and update the configuration of our OnDemand service to keep your communications safe.