Scalability testing for online assessments
Posted by Steve Lay
Last year I wrote a series of blog posts with accompanying videos on the basics of setting up virtual machines in the cloud and running them ready to install Questionmark Perception:
- Installing Perception in Amazon’s Cloud – Part 1
- Installing Perception in Amazon’s Cloud – Part 2
- I wandered lonely as a virtual machine – with apologies to William Wordsworth
This type of virtual machine environment is very useful for development and testing; we use a similar capability ourselves when testing the Perception software as well as new releases of our US and EU OnDemand services. One thing these environments are particularly useful for is scalability testing.
Scalability can be summarised as the ability to handle increased load when resources are added. We actually publish details of the scalability testing we do for our OnDemand service in our white paper on the “Security of Questionmark’s US OnDemand Service”.
The connection between scalability and security is not always obvious, but application availability is an important part of any organisation’s security strategy. For example, a denial-of-service or DoS attack is one in which an attacker deliberately exploits a weakness of a system in order to make it unavailable. Most DoS attacks do not involve any breach of confidentiality or data integrity, but they are still managed under the umbrella of security. Scalability testing focuses on the ‘friendly’ threat from increased demand but, like a DoS attack, the impact of a failure on the end user is the same: loss of availability.
As the popularity of our OnDemand service continues to increase, we’ve been ramping up our scalability testing, too. Using an external virtual machine service we are able to temporarily, and cost-effectively, simulate loads that exceed the highest peaks of expected demand. As more and more customers join our OnDemand service, the peaks of demand tend to smooth out when compared to a single customer’s usage — allowing us to scale our hardware requirements more efficiently. Our test results are also used to help users of Question Perception, our system for on-premise installation, provision suitable resources for their peak loads.
I thought I’d share a graph from a recent test run to help illustrate how we test the software behind our services. These results were obtained with a set of virtual resources designed to support a peak rate equivalent of 1 million assessments per day. The graph shows results from 13 different types of test, such as logging in, starting a test, submitting results, etc. The vertical axis represents the response times (in ms) for the minimum, median and 90th percentile cases at peak load. As you can see, all results are well within the target time of 5000ms.
I hope I’ve given you a flavour of the type of testing we do to ensure that Questionmark OnDemand lives up to being a scalable platform for your high-volume delivery needs.