Certification in the Cloud and the Move to Online Proctoring: An interview with SAP’s manager of global certification

John Kleeman Headshot

Posted by John Kleeman

I recently interviewed Ralf Kirchgaessner, SAP’s manager of global certification, about how the cloud is changing SAP certification. This is a shortened version of my conversation with Ralf. To read the full previously published post, check out this SAP blog.

John: What are the key reasons why SAP has a certification program?

Ralf: The overall mission of the program is that every SAP solution should be implemented and supported ideally by a certified SAP resource. This is to ensure that implementation projects go well for customers, and to increase customer productivity while reducing their operating costs. Customers value certification. In a survey of SAP User Group customers in Germany and the US, 80 percent responded that it was very important to have their employees certified and over 60 percent responded that certification was one of the criteria used to select external consultants for implementation projects.

John: What important trends do you see in high tech and IT certification?

Ralf: What comes first to the mind is the move to the cloud. Throughout the technology industry, the cloud drives flexibility and making everything available on demand. One aspect of this is that release cycles are getting quicker and quicker.

For certification, this means that consultants and others have to show that they are always up to date and are certified on the latest release. It’s not enough to become certified once in your lifetime: you have to continually learn and stay up to date. But of course if you are taking certification exams more often, certification costs have to be much lower. In some regions, people have to travel large distances to get to a test centre. With more frequent certification, it’s not practical to travel to a testing centre every time you take a certification. So our aim is to allow certification anytime and anywhere using the cloud.

John: How does online proctoring work for the candidate?

Ralf: A remote proctor monitors the candidate via a webcam, and there are a lot of security checks done by the proctor and by the system. For example, a secure browser is used, the candidate has to do a 360 degree check of his or her room, and there are lots of specific controls. For instance, you aren’t allowed to read the questions silently with your lips in case someone is watching or listening.

The great advantage to the candidate is flexibility. If someone says, “I’d like to do my exam in the middle of the night or on weekends because during the week I’m so busy with my project,” they can. They might say that they’d like to do their exam on Saturday afternoon: “After spending two hours playing with my kids, I’m relaxed to do my exam!” It’s such a flexible way to get certified and to quickly demonstrate that they have up-to-date knowledge and are allowed to provision customer systems.

John: Who benefits from certification in the cloud? Candidates, customers, partners or SAP?

Ralf: Of course, I think all benefit! Candidates have flexibility and lower cost. Customers can be sure that partner consultants who work for them are enabled and up to date. For partners, it’s a competitive advantage to show that their consultants are up to date, especially for new technologies like S/4HANA and Simple Finance. A partner is much more likely to be chosen to deploy new technologies if they can demonstrate that they have several consultants already certified in something that’s just been released. And for SAP, our goal is to have engaged consultants, happy partners and lower support costs. So everyone genuinely benefits.

John: What are some of the challenges?

Ralf: One example is that it’s important in cloud certification to get data protection right. SAP have very detailed requirements that we ensure our vendors like Questionmark meet.

Security is also a challenge. You need to prevent cheating and stealing questions.  And interfaces and integration need to be right. We have worked out how we get the data from our HR systems, how people book and subscribe to exams and then how they can authenticate with single sign-on into the certification hub to take cloud exams.

The delta concept also gives challenges. You need very precise pre-requisite management logic, where the certification software checks for example that, if you want to take the delta exam, you have already passed the core exam. It also can sometimes be difficult to prepare a good delta exam, particularly if a new release has very specific or detailed features, including some that apply in only some industries.

Lastly, providing seamless support is a challenge when using multiple vendors. The candidate doesn’t care where a problem happened: he or she just wants it fixed.

John: Where do you see the long term future of high-tech certification? Will there still be test centres, or will all certification be done via the cloud?

Ralf: Test centres won’t disappear at once, but there is a trend of moving from classroom-based learning and testing to learning and certification in the cloud. The future will belong to anytime, anywhere testing. The trend is for test centre use to decline, but it won’t happen overnight!

John: If another organization is thinking of moving towards certification in the cloud, what advice would you give them?

Ralf: Ensure that you are aware of the challenges I mentioned and can deal with them. And do some pilots before you try to scale.

Interested in learning more about Online Proctoring? I will be presenting a session on ensuring exam integrity with online proctoring at Questionmark Conference 2016: Shaping the Future of Assessment in Miami, April 12-15. I look forward to seeing you there! Click here to register and learn more about this important learning event.

Where do you deliver assessments from in a post-PRISM world?

John Kleeman HeadshotPosted by John Kleeman

Like many of you, I have been watching with interest revelations about government Internet surveillance initiatives. Technologically and legally, none of it is surprising. Businesses and governmental organizations around the world have frequently expressed concerns about the data privacy implications of the US Patriot Act.  Indeed, many of our customers cite data protection issues as factors in their decisions to opt for the Questionmark OnDemand service based at our European data centre.

Practically, I am torn between admiring our governments defending us against terrorism and pondering Benjamin Franklin’s saying that if you give up liberty for security, you lose liberty.

Wherever you stand on this issue, there are still questions to address about the practical implications this data protection challenge poses for those delivering assessments.  I thought it might be helpful to look at a couple of different scenarios and suggest data protection requirements you might look for when running assessments over the Internet.

Scenario 1. A US company looking for a safe place to deliver assessments from the Cloud

US flagSuppose you are a US company seeking to test your employees via a SaaS vendor. Suppose most employees are in North America but a few are spread round the globe. Here are the likely key data protection requirements:

1. Contract with a US service provider with confidentiality clauses.

2. Data centre and assessment results located in the US.

3. Data centre certified and audited to SSAE 16, the expected standard for quality data centres in North America.

4. Service provider and data centre operator certified under the U.S. Department of Commerce’s Safe Harbor Framework. This means they promise to comply with European data protection rules for data coming from Europe. Without this, you will have HR challenges testing your employees in Europe. With a lot of testing in Europe, you may want to look for stronger measures than Safe Harbor – see the White Paper (complimentary with registration): Responsibilities of a Data Controller When Assessing Knowledge, Skills and Abilities.

5. Vendors must have strong IT security including the latest SSL/TLS encryption and other technical measures.

Scenario 2. A European organization who wants to run assessments and keep data in Europe

European Union flagMany European companies or universities have a legal need to follow European data protection law and keep their data in Europe, and some may have constitutional requirements to avoid US oversight. Here are some of the key things they would look for:

1. Contract with a European service provider with confidentiality and data protection clauses.

2. Data centre with assessment results and personal data located inside the European Union.

3. Data centre certified and audited under ISO 27001, the expected standard for quality data centres in Europe.

4. This alone is only part of the story. The service provider and the data centre operator must not just be located in Europe, they must be European owned and not a subsidiary of a US company. If a US company runs a data centre or service in Europe, even if they run a subsidiary in Europe, they are required to hand over data on request to the US government, even if that data is in Europe. So if you work with a European subsidiary of a US LMS, VLE or other SaaS company, your data may be obtained by US enforcement agencies. According to a recent report by Reuters, a US judge has ruled that:

Internet service providers such as Microsoft Corp or Google Inc cannot refuse to turn over customer information and emails stored in other countries when issued a valid search warrant from U.S. law enforcement agencies

5. Again, all the legal data protection needs to be accompanied with good IT security. See our security comparison document for some questions to ask.

White Paper (complimentary with registration): Responsibilities of a Data Controller When Assessing Knowledge, Skills and Abilities.

Questionmark can meet both these needs. You can visit our website to learn how Questionmark OnDemand — US-based or EU-based — offers trustable solutions for either of these scenarios.

Celebrating 25 years of change — from DOS to SaaS

Question Mark for DOS (1988)John Kleeman HeadshotPosted by John Kleeman

Considering all the security, availability and flexibility we can achieve today with cloud-based assessment management systems, it’s remarkable to look back at the many changes and milestones we’ve seen over the past 25 years.

I wrote the first version of Question Mark for DOS in 1987-88. When I launched the company, 25 years ago, in London in August 1988, I always wanted to bring the benefits of computerized assessment to the world, but it was hard to foresee the dramatic technological changes that would transform our industry and make online assessment as widespread as it is today.

Coinciding with the rise of the PC, Question Mark for DOS empowered trainers and teachers to create, deliver and report on computerized assessments without having to rely on IT specialists.

Question Mark Designer for Windows (1993)QM Web 1995Things have been changing quickly ever since.  The early 1990s brought the move from DOS — functional but boring — to Windows — visual and graphical. This was radical at the time. To quote our marketing for Question Mark Designer for Windows, launched in 1993:

“Using Question Mark Designer, you can create tests using the full graphical power of Windows. You can use fonts of any size and type, and you can include graphics up to 256 colours. One of the most exciting features is a new question type, called the “Hot spot” question. This lets the student answer by “pointing” at a place on the screen.”

The switch to a visual user interface was huge, but the biggest paradigm shift of all was the move to delivering assessments over the Internet.

Pre-Internet, communicating results from assessments at a distance meant sending floppy disks by post. The World Wide Web made it possible to put an assessment on a web server, have participants answer it online and get instantly viewable results. This changed the world of online assessments forever.

QuestionmQuestion Mark Perception (1998)ark Technical Director Paul Roberts, who still plays an important role in Questionmark product development, wrote the code for the world’s first-ever Internet assessment product, “QM Web”, in 1995.  We followed up QM Web with the first version of Questionmark Perception, our database-driven assessment system, in 1998.

Eric Shepherd founded the U.S. division of Questionmark in the 1990s and in 2000 became CEO of what is now a global company. He is the heart and soul of Questionmark, an inspiring chief executive who has turned Questionmark from a small company into an industry leader.

One key paradigm shift in the 2000s was the desire to use surveys, quizzes, tests and exams in more than one department — across the entire enterprise. To make this practical, we began building scalability, reliability, translatability, accessibility, maintainability and controllability into our technologies. These attributes, along with multiple delivery options and in-depth reporting tools, are key reasons people use Questionmark today.

Cutting the ribbon at the Questionmark European Data Center

Opening our European Data Centre last year marked a major expansion of our cloud-based Questionmark OnDemand service

In recent years, we’ve seen another dramatic change – towards software-as-a-service applications in the “cloud”.  Just as Question Mark for DOS 25 years ago empowered ordinary users to create assessments without needing much IT support, so Questionmark OnDemand today allows easy creation, delivery and reporting on assessments without in-house servers.

So what’s in store for the future? Technology is making rapid advances in responsive design, security, “big data”, mobile devices and more. Questionmark keeps spending around 25% of revenues on product development. The huge demand for online assessments is making this our busiest time ever, and we expect continued, rapid improvement.

I’d like to thank our customers, suppliers, partners, users and employees – whose support, collaboration and enthusiasm have been critical to Questionmark’s growth during our first 25 years. I look forward to continuing the journey and am eager to work with all of you to shape what happens next!

Hard and soft defences in our castle in the cloud

John Kleeman HeadshotPosted by John Kleeman

How do you communicate the security of a service like Questionmark OnDemand? I find the concept of a castle useful in explaining security. Back in medieval days, people stored safe things – their “crown jewels” – inside a castle to protect them. And today, websites that store confidential information need to set up a “castle in the cloud” to protect data.

Let’s look at a castle’s defences:  hard defences such as walls moats — and soft defences such as the guards who man the watch towers and entry points:




How do a castle’s hard and soft defences translate into defences for software-as-a-service in the Cloud?

Hard defences

A castle has a moat and layers of walls. Questionmark OnDemand has firewalls,  and it is tiered so data moves from presentation tier to business tier to data tier to protect the data. (See What’s the key to reliable assessment management? for more on tiers.)

And a castle has watch towers. Questionmark OnDemand has intrusion detection: automatic systems that keep watch for inappropriate traffic.

A castle also has limited entry points. Questionmark OnDemand has limited entry points, too, and it only lets certain types of Internet traffic come in, for example all browser traffic has to use a sufficient level of HTTPS.
Soft defences

A castle is only strong if it has alert guards to protect it. In the medieval world, you needed trained guards on duty 24/7 in case of intruders. You also needed to carefully check identity and authorization in case someone came in to steal the crown jewels by deceit.

Similarly, in a service like Questionmark OnDemand, we have authentication and authorization systems. We also have people behind the scenes who are trained to detect security risks. For instance, everyone at Questionmark is trained on data security and has to take and pass a data security test each year.

I hope the castle analogy amuses and informs. But of course, the real point is that your assessment data is well protected in our castle.

For more information on the security of Questionmark Ondemand, see our security white paper. Watch this blog for future articles on security – or in the meantime, feel free to check out my earlier post 13 Scary Questions to Ask your Assessment Cloud Provider.


eAssessment in the Cloud, Sunshine or Thunderstorm?

Sunshine or Thunderstorm?Posted by John Kleeman

Earlier this week, I presented at the online part of the eAssessment Scotland conference on the advantages and disadvantages for academic institutions of using eAssessment in the Cloud “on-demand” or installing it “on-premise” within the institution. Does an on-demand eAssessment service give continual sunshine to a university or college? Or is it safer to install it locally and go on-premise? What questions do you need to ask about the potential thunderstorms using the Cloud?

Questionmark offers both Questionmark Perception, an installable assessment management system, and Questionmark OnDemand, our scalable software-as-a-service system, so we can see the pros and cons of both approaches- and can offer some unbiased advice.

Here is the presentation I gave – you can see it embedded at the end of the post or else view it on the Questionmark Slideshare site.

The presentation suggests that for a university or college, on-demand may be stronger in these areas:

  • Access to innovation
  • Speed/flexibility of deployment
  • Reliability and uptime
  • Scalability
  • Security and cheating
  • Getting IT bandwidth

And that 0n-premise may be stronger in these:

  • Ease of customization/integration
  • Connectivity
  • Governments accessing your data

In these areas, you need to look into the details to determine what would work best in your situation:

  • Data protection
  • Can you change providers?
  • Costs, features and other factors

I believe that for a lot of universities and colleges on-demand offers a lot of value. This is especially so if their IT department is focused elsewhere and does not easily have the bandwidth to manage eAssessment. But it is very important to get your solution right, and if you’re looking at On-demand, you might like to read a paper I presented at the 2012 International Computer Assisted Assessment Conference on How to Decide between On-demand and On-premise eAssessment. This includes a lot of useful questions to ask potential providers when evaluating potential on-demand solutions. You can see the paper here.

I hope you find the presentation useful.

13 Scary Questions to Ask your Assessment Cloud Provider

Posted by John Kleeman

As its Halloween I thought you might enjoy learning about 13 questions that might scare your Assessment Cloud provider.

Let me first share some background information …13 Scary Questions to Ask your Assessment Cloud Provider

With increasing use of Cloud systems like Google Docs, Microsoft’s Office 365, and Amazon, and with enterprise software giants like Oracle and SAP offering OnDemand services, many organizations that previously managed IT internally are delegating the running of servers. A Cloud service can save you money, and allow you to focus on core business and user issues, by letting someone else deal with the technology.

Secure and scalable Assessment Clouds are the next wave of tools available that help organizations to measure knowledge, skills, and attitudes securely for certification, regulatory compliance and successful learning outcomes
As you consider moving your assessments to a Cloud, you need to ensure your provider is offering the best possible service, security and data protection. You want a provider who is fully invested in giving strong security, scalability, elasticity and robustness, not just someone running a server under a desk! Exam security has different challenges and demands to other kinds of IT due to the confidentiality of personally identifiable information, questions and results, so you need to make sure that the system you use is safe and secure.

Here are 13 questions you might scare the less professional Assessment Cloud providers in the marketplace:

1. Do you host assessments in a well-established Data Center, certified to SAS 70 Type II, SSAE 16 Type II or ISO 27001?

2. Does your Data Center have multiple connections to the power grid with onsite generators with at least 24 hours fuel onsite in case of power outages?

3. Does your Data Center have multiple, fast Internet links so that if one goes down, connectivity remains?

4. Is every server in the system load balanced and does every component have redundancy, so that if any one system fails, another can take over?

5. Is browser access to assessments and administration protected by SSL (or TLS) to 128 bits or higher, so that assessment data and results cannot be intercepted on the Internet?

6. Do you follow industry good practice in software development to reduce surface areas of attack and protect against security vulnerabilities? Common methodologies to work with are called STRIDE and DREAD.

7. Do you have separate development/integration areas and staging areas to test on before deploying to production?

Questionmark’s OnDemand Testing and Deployment Process
Questionmark’s OnDemand Testing and Deployment Process

8. Do you have a data security policy for your employees who run the service to ensure that they maintain the secrecy of customer data? Does the policy include confidentiality agreements, background checks on employees, regular training, and regular testing of employees to check they that understand data security?

9. Can I see information on real time information on the current status and uptime, and access statistics from round the world? See status.questionmark.com for an example of what you might look for from a provider.

10. Is the service monitored and run 24/7 at both Data Center, network, hardware and application level, so that problems out of hours will be fixed?

11. Are results data backed up safely at least once an hour, so that in the event of a catastrophe, you should never lose more than an hour’s worth of data?

12. What access might government agencies have to data of foreign nationals and are your systems Safe Harbour Certified?

13. What is your track record do you have for being a trustworthy provider with references and case studies to back your claims up?

The answers to these questions for Questionmark’s OnDemand Service are all yes. If you want to find out more, read more details in our new white paper, Security of Questionmark’s OnDemand Service available here.