Scalability testing for online assessments

Posted by Steve Lay

Last year I wrote a series of blog posts with accompanying videos on the basics of setting up virtual machines in the cloud and running them ready to install Questionmark Perception:

This type of virtual machine environment is very useful for development and testing; we use a similar capability ourselves when testing the Perception software as well as new releases of our US and EU OnDemand services. One thing these environments are particularly useful for is scalability testing.

Scalability can be summarised as the ability to handle increased load when resources are added. We actually publish details of the scalability testing we do for our OnDemand service in our white paper on the “Security of Questionmark’s US OnDemand Service”.

The connection between scalability and security is not always obvious, but application availability is an important part of any organisation’s security strategy. For example, a denial-of-service or DoS attack is one in which an attacker deliberately exploits a weakness of a system in order to make it unavailable. Most DoS attacks do not involve any breach of confidentiality or data integrity, but they are still managed under the umbrella of security. Scalability testing focuses on the ‘friendly’ threat from increased demand but, like a DoS attack, the impact of a failure on the end user is the same: loss of availability.

As the popularity of our OnDemand service continues to increase, we’ve been ramping up our scalability testing, too. Using an external virtual machine service we are able to temporarily, and cost-effectively, simulate loads that exceed the highest peaks of expected demand. As more and more customers join our OnDemand service, the peaks of demand tend to smooth out when compared to a single customer’s usage — allowing us to scale our hardware requirements more efficiently. Our test results are also used to help users of Question Perception, our system for on-premise installation, provision suitable resources for their peak loads.

I thought I’d share a graph from a recent test run to help illustrate how we test the software behind our services. These results were obtained with a set of virtual resources designed to support a peak rate equivalent of 1 million assessments per day. The graph shows results from 13 different types of test, such as logging in, starting a test, submitting results, etc. The vertical axis represents the response times (in ms) for the minimum, median and 90th percentile cases at peak load. As you can see, all results are well within the target time of 5000ms.

I hope I’ve given you a flavour of the type of testing we do to ensure that Questionmark OnDemand lives up to being a scalable platform for your high-volume delivery needs.


One Response to “Scalability testing for online assessments”

  1. Steve Lay Steve Lay says:

    One of the questions I’ve been asked about this post is what the various tests represent. Obviously these results are from just one test run for just one target load but I’ve tried to pick figures that are representative. Tests 3 and 10 correspond to starting an assessment (block) and tests 4 and 11 correspond to submitting answers and displaying the resulting score. The other bars represent other actions, such as logging in, navigating, and so on. I hope that helps clarify the graphic shown.

Leave a Reply