SAP to resell Questionmark software to its customers

John Kleeman HeadshotPosted by John Kleeman

Yesterday we were very happy to  announce the signing of a global reseller agreement with SAP AG, under which SAP will resell our technologies to its customers. You can see the press release here.

Under the agreement, SAP will resell assessment technologies from Questionmark under the name SAP® Assessment Management by Questionmark.
SAP® Assessment Management by Questionmark will enable SAP customers to create, deliver and analyze surveys, quizzes, tests and exams. The software  will complement SAP Learning Solution and SuccessFactors Learning, allowing use of assessments for purposes such as certification, regulatory compliance, and health and safety training for a company’s internal tracking purposes.

SAP will deliver the application on-premise and as a cloud offering to give customers a secure, collaborative environment for creating learning assessments in multiple languages and delivering to a wide variety of browsers and devices. We then anticipate that SAP® Assessment Management by Questionmark users will be able to evaluate results and inform stakeholders with timely, meaningful analyses and reports on items and tests.

This agreement builds on our established relationship as an SAP software solution and technology partner, as well as the longstanding compatibility of our assessment management software with both SAP Learning Solution and SuccessFactors Learning.

I am delighted that this agreement will help SAP customers get even more value from their investments in enterprise learning software. I’ve been working closely with SAP as they have reviewed our technology and I admire SAP’s commitment to quality in learning and technology and the strength and quality of their team.

I’m pleased to share a quote within the press release from Markus Schwarz, SVP and global head of SAP Education:

“Since the need to assess learning is at an all-time high, the addition of this offering to our portfolio of collaborative learning software from SAP Education is well-timed indeed. And because it works together with SuccessFactors Learning and SAP Learning Solution, the new SAP Assessment Management application by Questionmark will dovetail perfectly with our strategy to bring cloud-based learning to SAP customers and partners worldwide.”

Summer webinars — including tips on better test planning and delivery

Joan Phaup HeadshotPosted by Joan Phaup

Students (and teachers) may be clicking their heels about summer vacation, but the joy of learning continues year-round for us!

Helping our customers understand how to use assessments effectively is as important to us as providing good testing and assessment technologies — we ‘re keeping our web seminars going strong during the summer months.

Here’s the current line-up:

Questionmark Customers Online: Using Questionmark and SAP for Effective Learning and Compliance — June 20 at 1 p.m. Eastern Time:

Learn about the use of Questionmark and SAP for a wide array of learning and compliance needs, including safety training, certifications and regulatory compliance testing. This presentation by Kati Sulzberger of BNSF Railway also describes how Questionmark helped the company meet some unique test delivery requirements.

Five Steps to Better Tests: Best Practices for Design and Delivery — July 18 at noon Eastern Time:

Get practical tips for planning tests, creating items, and building, delivering and evaluating tests that yield actionable, meaningful results. Questionmark Product Owner Doug Peterson, who will present this webinar, previously spent more than 12 years in workforce development.  During that time, Doug created training materials, taught in the classroom and over the Web, and created many online surveys, quizzes and tests.

Questionmark Customers Online: Achieving a Better Assessment-Development Process — August 22 at 1 p.m. Eastern Time:

Need a better assessment building process? Find out how enterprise architecture principles can help you and your team work more efficiently.  Tom Metzler,  Knowledge Assessment Administrator at TIBCO Software, Inc.,  will explain how the company’s certification team uses well-established software architecture principles to continually improve the efficiency of its assessment development process. Find out how using systematic processes and thorough documentation result in better information for subject matter experts, time-savings and higher-quality assessments.

Introduction to Questionmark’s Assessment Management System — Choose from a variety of dates and times

This primer  explains and demonstrates key features and functions available in Questionmark OnDemand and Questionmark Perception. Spend  an hour with a Questionmark expert learning the basics of authoring, delivering and reporting on surveys, quizzes, tests and exams.

Click here for more details and free online registration.

On-demand or on-premise: Which is better for talent management?

John Kleeman HeadshotPosted by John Kleeman

Is it better to run assessment, learning and other talent management software on-demand, in the Cloud? Or is it wiser to run software on-premise, within your organization’s firewall?

I recently wrote about this on the SAP community and received a lot of feedback; I‘d like to share the topic with readers of the Questionmark Blog.

In this post I will share 6 reasons why the Cloud is usually better. And in my next post I’ll give you 4 reasons why it may not be.

1. On-demand gives you access to innovation and use of mobile devices


A critical advantage of on-demand deployment – or software as a service (SaaS) — is that you get the latest version of software. Most providers upgrade all their customers at the same time to the latest version, and you get bug fixes, feature improvements, security fixes and innovation as part of the service. With on-premise, you are in control of when you install updates. But due to the resources required to upgrade, it’s commonplace to only upgrade once every year or two, and therefore be several versions behind an on-demand system. Support for the latest mobile devices is an obvious casualty..

To quote Ed Cohen of SuccessFactors:

“If you look at the rate of innovation that can occur with a SaaS product as against a company maintaining a behind the firewall instance of something, it becomes super important for learning and talent.”


2. Deployment is easier with on-demand and allows quick pilots

An on-premise system needs setup of servers and software installation. This takes planning, time and resources, whereas an on-demand system can usually be deployed within hours of ordering it. An on-demand system is also easier to scale up and expand. You can start small with one project and add users or departments as needed.

3. On-demand requires less corporate IT bandwidth

This is often the strongest reason to go on-demand in the learning and assessment space. Corporate IT departments are typically overloaded, and talent management software is not their top priority. This creates a bottleneck, which in turn delays deployment.

On-demand still needs the involvement of corporate IT, but you can usually make headway and provide improved functionality quicker than when deploying on-premise.

4. You don’t need to worry about scalability with on-demand

With an on-premise solution, you have to scale servers to cope with the busiest times (e.g. an end-of-year deadline, exam season or a compliance milestone). But if you use on-demand software, you delegate this to the Cloud provider, who will usually be able to expand to handle your highest load.

5. On-demand is easier to make secure

Both on-premise and on-demand can be very secure, but achieving a high level of security is expensive and involves constant vigilance. Unless you invest heavily in security, Cloud providers will usually provide higher security than the typical on-premise solutions.

This point is well described by SAP’s  Prashanth Padmanabhan in his blog article Why Do We Keep Our Valuables In A Bank Locker?. He states:

“… one of the SAP – SuccessFactors Hybrid customers announced publicly that their own security audit found that SuccessFactors cloud infrastructure was more secure than their own fire wall.”

And the respected UK Universities and Colleges Information Systems Association says in its Cloud briefing paper:

“In practice, data is probably more secure in cloud services than can be provided by in house solutions.”

6. On-demand is usually more reliable

Usually, providing your users have good Internet connectivity, an on-demand system will also be more reliable and have higher up-time.

Stylized picture of bridge

Unless you invest heavily in your on-premise infrastructure, a professionally maintained on-demand server is likely to provide a higher level of 24/7 availability and uptime than a locally maintained system. A professional system is likely to have redundancy in every component and will not fail if a piece of hardware fails, whereas it may not be cost-effective to have such redundancy in an on-premise system. Redundancy makes sure, just like in a bridge over a river, that if one piece fails, the rest of the bridge survives.

In a follow-up post, I’ll explain some reasons why on-premise can be better.

Crab cakes, customers and cool code in Charm City

John Kleeman HeadshotPosted by John KleemanHarbor View

I’ve just back from the Questionmark user conference in Baltimore “Charm City”, famous for its delicious crab cakes.

The best crab cakes let the flavour of crab shine through the other ingredients that hold the cake together. Similarly, with a conference, the main ingredient is the learning we take home with us. The other things that hold the event together – the networking, social events and discussions that take place – make the learning very enjoyable.

Here are six trends I observed last week:

1. Scaling up works well and is commonplace. Many Questionmark users have high volumes without drama. One case study at the conference told of a company deploying Questionmark with SAP in two months to deliver safety qualification assessments to 35,000 employees. And another told of delivering 2.3 million Questionmark assessments in 2012.

2. Authoring improvements. Our recent web-based authoring improvements are exciting users. In the 25 years we’ve been operating, Questionmark software has always been about enabling users to easily create, deliver and report on tests, quizzes, surveys and exams. There is a lot of excitement about what we’ve introduced recently in web-based authoring of questions and assessments, especially the way people can easily collaborate and see and review changes in items.

3. Acceptance of the Cloud. Some organizations prefer to deploy on-premise, others deploy in the Cloud with Questionmark OnDemand, and we support both. With other HR systems moving to the Cloud, more Questionmark customers are looking at moving to OnDemand. One influence is that SAP is telling all its HR customers that the future for them is in the Cloud with SuccessFactors; people are thinking “if you’re using SAP in the Cloud, then it makes sense to use Questionmark there, too”.charles networking

4. 70:20:10 is a given, and people are learning how to use it. Charles Jennings keynote was not without its controversies – many people still believe in the Kirkpatrick and Phillips models of learning evaluation, but he suggests levels 1, 2 and 5 have dubious validity in today’s learning environment. But there is widespread acceptance that the 70:20:10 model makes sense in corporate learning – that we learn 70% of what we learn on the job, 20% from others and only 10% from formal learning. There are questions around how you assess workplace learning – observational assessments are many people’s answer, and how you combine 70:20:10 with mandatory compliance training and ensuring people learn to work safely.

5. New methods of secure delivery matter. The availability of Questionmark Secure for use with Macs is making it easier for organizations, especially universities/colleges, to use a secure browser for delivering exams. People are also starting to use remote proctoring via organizations like our partner, ProctorU. Mobile assessment delivery is also now real not just being experimented with.

6. Cool code makes a difference. There’s been a lot of talk in recent years about how HTML 5 will make a difference to user interfaces and it’s great to see this happening for real. Questionmark is using HTML 5 to make our authoring and delivery interfaces easier and more natural. See for example this video: How to create hotspot questions for quizzes and tests. Code like this is exciting our customers and is going to provide very compelling web interfaces moving forwards.

I hope you enjoyed this and our other blog entries about the conference. We do our best to share about Questionmark and assessment remotely, but if you have a chance to attend one of our US or European user conferences in person, I promise you will learn and gain value.

How SAP use Questionmark to deliver certification exams in 20 languages

Posted by John Kleeman

At the recent European Association of Test Publishers conference, I presented jointly with Sue Martin and Ralf Kirchgaussner of SAP on blended delivery within SAP’s certification programme. You can see our presentation on SlideShare, but I thought I’d highlight in this blog how SAP uses Questionmark’s Translation Management System to organize its translation of certification exams in up to 20 languages.

SAP sees certification as an enabler for project success and quality. There is evidence in the IT industry (see for example here) that functional performance improves when a majority of a project team are certified. So for SAP, certification is part of their strategic need to ensure successful use of their technology. SAP are also a genuinely global company; and if it needs to be done, it needs to be done for people in a variety of geographies and languages.

Languages used in the SAP certification programmeAs you can see in the Languages graph on the right, the most popular language for SAP certifications is English. But many are delivered in other languages with German, Spanish, Portuguese and Japanese the most widely used non-English languages.

SAP translate items and assessments using Questionmark’s Translation Management System, exporting the text to XML for translation by professional translators. They then import the translated text back into the Questionmark Perception assessment management system for review and delivery.

See the screenshot below showing a topic in the SAP repository translated into 18 languages:

Questionmark Perception Translation Management System

I hope you enjoyed this glimpse into how SAP use translated assessments. You can see more about SAP’s certification programme and how they are looking to move into a more blended delivery environment in our presentation which is embedded below:

It’s one thing to train, another to prove training

Simone Buchwald Posted by John Kleeman

I recently published an interview in the SAP Community Network with Simone Buchwald, who was the product manager of SAP’s Learning Solution (LMS) when it first came out and now works in the SAP Ecosystem as a consultant at EPI-USE. She shared some information on assessments within SAP and trends in compliance, which I thought readers of this blog might enjoy, here is an extract from the interview:


Why do people use assessment in the SAP Learning Solution (LSO)?

Assessments play a vital role in LSO as they broadly do in every LMS, especially as a big focus for LSO use is compliance management. Assessments are a crucial part of a compliance solution, because organizations have to test and document that people understand what they are supposed to learn.

I remember when I was at SAP and we made contact with Questionmark – we immediately decided that we would not extend the LSO test authoring system much, but would instead suggest Questionmark as the assessment partner tool for LSO because of its strong capabilities.


What is the best way to document training for compliance purposes. Is it okay just to record that someone has been through training or do you need to assess to check understanding?

I see a trend both in regulated and non-regulated environments that it is not enough just to track the completion of training, you also need to validate the understanding of learning.

So I think it is important to check understanding by assessments, both for tracking purposes and in case an auditor wants to see the records. It is one thing to do something, and another to prove it. This even extends nowadays outside of the core regulatory use cases to courses where the company wants to track the completion and the understanding, for example  in sales training to prove the value of the learning for the company.


What trends do you see in compliance; how are things changing?

A lot of the compliance processes that we have seen in the past that have been mandatory in compliance-driven businesses —  like pharmaceuticals, chemicals and mining — we are now seeing also expanding in other industries and customers. For instance we see needs for mandatory training of certain business processes, sometimes imposed by regulation, but sometimes just imposed by the business itself. For example, when organizations roll out equal opportunities training, the company may decide to implement the same process as for compliance-required training – checking who has completed it, checking who has done the assessments and been validated. And often this is driven from internal business needs, not external regulators.

People are seeing the advantage of conducting learning and assessments not just when the regulator requires it, but for their own business purposes.


You can see the full interview here.