How is the SAP Global Certification program going? A re-interview with SAP’s manager of global certification, part 2.

Posted by Zainab Fayaz

This is the second part of the two-part interview between, John Kleeman, Founder and Executive Director at Questionmark and Ralf Kirchgaessner, Manager of the SAP Global Certification program. This is a continuation of the use of Questionmark software in their Certification in the Cloud program. You can read the first part here. In the second part of the interview, John asks questions about the business benefits of certification and what advice Ralf has for other organizations.

John: What are the business benefits to SAP of certification?

Ralf: There are many benefits to the SAP Global Certification. So, let’s begin from the individual learner’s perspective.

Firstly, earning the SAP Global Certification increases your personal value; not only does it drive personal development; which often leads to increased responsibilities and promotion within your organization, but it also showcases and proves that you stay current and update your skills to the latest releases. Additionally, since 2018, professionals can gain wider recognition through sharing their SAP Global Certification digital badges.

SAP Global Certification is of great value not only for individuals but also for consultancies in the SAP ecosystem. SAP Global Certifications provide a clear measure of a company’s organizational capabilities, which give a competitive advantage, especially if the company has certified professionals in new and innovative areas, like SAP C/4HANA Cloud.

John: What about the customers? What benefits are there for them?

Ralf: Indeed, the most important benefit is the value for our customers. If SAP can ensure that the consultancy eco-system is well enabled and certified, it helps reduce the total cost of ownership (TCO) and ensures successful implementation costs. And in the end, this is of course also important for SAP, as this helps to increase the adoption of our software and reduces implementation risks.

John: Tell me a bit more about the recently introduced digital badges for people who get certified that you just mentioned. How useful is that?

Ralf: The introduction of digital badges for SAP Global Certification has been an absolute success! Making your workforce visible on the market is important and by sharing the digital badge proves that the workforce is currently in their knowledge. If on LinkedIn, you search for ‘certified SAP consultants’, you would find thousands of shared badges. Digital badge claim rates beyond industry standards show that people waited with much anticipation to share their achievements digitally.

We are constantly looking for ways to improve our services and with the help of Questionmark, going forward we will be able to issue badges, even faster. In the near future, once candidates have passed their SAP Global Certification exam this will trigger the issuing of badges in “real-time”!

We have reached our ultimate goal and an overall mission of our certification programme if customers ask consultants for their digital badges to show their SAP Global Certification status.

John: There seems a slow move across the community from test centers to online proctoring. I know that for SAP, you deliver some exams in your offices but most in the cloud with online proctoring. How do you see this changing in the industry in general? Will all IT exams be done by online proctoring one day soon?

Ralf: SAP very much uses the model of taking exams wherever and whenever it is most convenient. Nevertheless, we use one harmonized infrastructure, for all our exams and these can be taken at our offices, in classrooms or in the cloud.

I think much of this evolves from the changing landscape in learning behaviors and offerings. In terms of the advantages of using test centers and online proctoring; there is a legitimate reason for test centres to exist; as there are groups of people who will still want to learn together – in one place at one time. However, as the shift moves towards a rise in remote learning, both synchronous (live virtual classrooms) and asynchronous, which are supported by social and peer learning via online learning rooms, then of course, online proctoring will become more popular.

John: What advice would you give to other high-tech companies who are thinking of setting up or improving their certification program?

Ralf: Two things instantly come to mind – online proctoring and digital badging. Certification programs that do not use online proctoring and digital badging should urgently consider improving their program as the benefits of implementing both features are tremendous.

More on certification
Interested in learning more about certification programs?  Find out how you can build your own certification program in 10-easy steps.

 

How is the SAP Global Certification program going? A re-interview with SAP’s manager of global certification, part 1.

Posted by Zainab Fayaz

Back in 2016, John Kleeman, Founder and Executive Director of Questionmark interviewed Ralf Kirchgaessner, Manager of SAP Global Certification program about their use of Questionmark software in their Certification in the Cloud program and about their move to online proctoring. You can see the interview on the Questionmark blog here. We also thought readers might be interested in an update, so here is a short interview between the two on how SAP are getting on three years later:

John: Could you give us an update on where you are with the Certification in the Cloud program?

Ralf: The uptake, adoption and increase of Certification in the Cloud is tremendous! Over the years we have seen a significant increase in the volume of candidates taking exams in the cloud; the numbers doubled from 2016 to 2017 and increased almost by 60% in 2018. This means more than 50% of SAP Global Certification exams are now done remotely!

John: Are all your SAP Global Certification exams now available online in the cloud?

Ralf: Nearly so. By mid-2019 we plan on having the complete portfolio of every SAP exam available on the cloud. This is great news for our learners who have invested in a Certification in the Cloud subscription. So, we then have Certification in the Cloud not only for SAP SuccessFactors and SAP Ariba, but for all products, including SAP C/4HANA.

John: How many different languages are your exams translated into?

Ralf: This depends on the portfolio. Some of our certifications are available in English and others, such as for SAP Business One are translated in up to 20 languages.

John: How are you dealing with the fast pace of change within SAP software in a certification context? How do you ensure certifications stay up to date when the software changes?

Ralf: This is of course a challenge. In previous years, it was the case of getting certified once every few years. However, now you must keep your skills up-to-date and stay current with quarterly release cycles of our SAP Cloud solutions. Also, for people who are first timers or newly enter the SAP eco-system; it is important that they are certified on the latest quarterly release.

To help overcome this challenge, we have developed an agile approach to updating our exams; we use the Questionmark platform for those who are new to the eco-system to help them getting certified initially. We also have a very good process in place and often use the same subject matter experts when it comes to keeping up to the speed of software changes.

For already certified professionals, another way to remain up to date is through our ‘Stay Current’ program. For some of our solutions, partners have to come back every 3 months to show that they are staying current. They do this in the form of taking a short “delta” knowledge assessment. For instance, for certified professionals of SAP SuccessFactors it is mandatory to stay current in order to get provisioning access to the software systems.

In 2018, SAP’s certification approach was acknowledged with the ITCC Innovation Award. Industry peers like from Microsoft, IBM and others recognized this great achievement with this award.

 

Can you be GDPR compliant without testing your employees?

Posted by John Kleeman

The GDPR is a new extra-territorial, data protection law which imposes obligations on anyone who processes personal data on European residents. It impacts companies with employees in Europe, awarding bodies and test publishers who test candidates in Europe, universities and colleges with students in Europe and many others. Many North American and other non-European organizations will need to comply.

See my earlier post How to use assessments for GDPR compliance for an introduction to GDPR. The question this blog post addresses is whether it’s practical for a large organization to be compliant with the GDPR without giving tests and assessments to their employees?

I’d argue that for most organizations with 100s or 1000s of employees, you will need to test your employees on your policies and procedures for data protection and the GDPR. Putting it simply, if you don’t and your people make mistakes, fines are likely to be higher.

Here are four things the GDPR law says (I’ve paraphrased the language and linked to the full text for those interested):


1. Organizations must take steps to ensure that everyone who works for them only processes personal data based on proper instructions. (Article 32.4)

2. Organizations must conduct awareness-raising and training of staff who process personal data (Article 39.1). This is extended to include “monitoring training” for some organizations in Article 47.2.

3. Organizations must put in place risk-based security measures to ensure confidentiality and integrity and must regularly test, assess and evaluate the effectiveness of these measures. (Article 32.1)

4. If you don’t follow the rules, you could be fined up to 20 million Euros or 4% of turnover. How well you’ve implemented the measures in article 32 (i.e. including those above) will impact how big these fines might be. (Article 83.2d)


So let’s join up the dots.

Firstly, a large company has to ensure that everyone who works for it only processes data based on proper instructions. Since the nature of personal data, processing and instructions each have particular meanings, this needs training to help people understand. You could just train and not test, but given that the concepts are not simple, it would seem sensible to test or otherwise check their understanding.

A company is required to train its employees under Article 39. But the requirement in Article 32 is for most companies stronger. For most large organizations the risk of employees making mistakes and the risk of insider threat to confidentiality and integrity is considerable. So you have to put in place training and other security measures to reduce this risk. Given that you have to regularly assess and evaluate the effectiveness of these measures, it seems hard to envisage an efficient way of doing this without testing your personnel. Delivering regular online tests or quizzes to your employees is the obvious way to check that training has been effective and your people know, understand and can apply your processes and procedures.

Lastly, imagine your company makes a mistake and one of your employees causes a breach of personal data or commits another infraction under the GDPR? How are you going to show that you took all the steps you could to minimize the risk? An obvious question is whether you did your best to train that employee in good practice and in your processes and procedures? If you didn’t train, it’s hard to argue that you took the proper steps to be compliant. But even if you trained, a regulator will ask you how you are evaluating the effectiveness of your training. As a regulator in another context has stated:

“”where staff understanding has not been tested, it is hard for firms to judge how well the relevant training has been absorbed”

So yes, you can imagine a way in which a large company might manage to be compliant with the GDPR without testing employees. There are other ways of checking understanding, for example 1:1 interviews, but they are very time consuming and hard to roll out in time for May 2018. Or you may be lucky and have personnel who don’t make mistakes! But for most of us, testing our employees on knowledge of our processes and procedures under the GDPR will be wise.

Questionmark OnDemand is a trustable, easy to use and easy to deploy system for creating and delivering compliance tests and assessments to your personnel. For more information on using assessments to help ensure GDPR compliance visit this page of our website or register for our upcoming webinar on 29 June.

Internet assessment software pioneer Paul Roberts to retire

Paul Roberts photoPosted by John Kleeman

We think of the Internet as being very young, but one of the pioneers in using the Internet for assessments is about to retire. Paul Roberts, the developer of the world’s first commercial, Internet assessment software is retiring in March. I thought readers might like to hear some of his story.

Paul was employee number three at Questionmark, joining us as software developer in 1989 when the company was still working out of my home in London.

During the 1990s, our main products ran on DOS and Windows. When we started hearing about the new ideas of HTML and the web, we realized that the Internet could make computerized assessment so much easier. Prior to the Internet, testing people at a distance required a specialized network or sending floppy disks in the mail (yes people really did this!). The idea that participants could connect to the questions and return their results over the Internet was compelling. With me as product manager, tester and documenter for our new product — and Paul as lead (and only!) developer — he wrote the first version of our Internet testing product QM Web, which we released in 1995.

QM Web manual cover

QM Web became widely used by universities and corporations who wanted to deliver quizzes and tests over the Internet. Later in the nineties, learning from the lessons of QM Web, we developed Questionmark Perception, our enterprise-level Internet assessment management system still widely used today. Paul architected Questionmark Perception and for many years was our lead developer on its assessment delivery engine.

One of Paul’s key innovations in developing Questionmark Perception was the use of XML to store questions. XML (eXtensible Markup Language) is a way of encoding data that is both human-readable and machine-readable. In 1997, Paul implemented QML (Question Markup Language) as an early application of this concept. QML allowed questions to be described independently of computer platforms. To quote Paul at the time:

“When we were developing our latest application, we really felt that we didn’t want to go down the route of designing yet another proprietary format that would restrict future developments for both us and the rest of the industry. We’re very familiar with the problems of transporting questions from platform to platform because we’ve been doing it for years with DOS, Windows, Macintosh and now the Web. With this in mind, we created a language that can describe questions and answers in tests, independently of the way they are presented. This makes it extremely powerful because QML now enables the same question database to be presented no matter what computer platform is chosen on or whatever the operating system.”

Questionmark Perception and Questionmark OnDemand still use QML as their native format, so that every single question delivered by Questionmark technology has QML as its core. QML was very influential in the design of the version 1 IMS Question & Test Interoperability specification (IMS QTI), which was led by Questionmark CEO Eric Shepherd and to which Paul was a major contributor. Paul also worked on other industry standards efforts including AICC, xAPI and ADL SCORM.

Over the years, many other technology innovators and leaders have joined Questionmark, and we have a thriving product development team. Most members of our team have had the opportunity to learn from Paul over the years, and Paul’s legacy is in safe hands: Questionmark will continue to break new frontiers in computerizing assessments. I am sure you will join me in wishing Paul well in his personal journey post-retirement.

Seven New Year’s Resolutions to Keep Your Assessments Safe

Paper with "Resolutions" written on it implying one is about to write some resolutions downJohn Kleeman HeadshotPosted by John Kleeman

Many blogs at this time of year seek to predict the year ahead, and many of them foresee more data breaches and security incidents in 2017.  But I’m a great believer that the best way to predict the future is to create or change it yourself. So if you want to reduce the chances of your assessment data security being breached in 2017, make some of the things you’ve talked about happen.

Here are some possible New Year’s resolutions that could help keep your assessments safe and secure.

1. Audit your user accounts. Go through each of your systems that hold or give access to assessment data, and check there are no accounts for ex-employees or ex-contractors. Make sure there are no generic or test accounts that do not belong to a current individual. Dormant accounts like this are a common route to a breach. Also check that no one who has changed role has the privileges of their old role.

2. Run an incident response table-top practice exercise. This is a session where you gather together those responsible for security, pretend there is a breach or other incident and work through verbally how you’d deal with it as a team. You can do this in a couple of hours with good preparation, and it allows you to check your procedures and ensure people know what to do. It will often give useful insight into improving your preparedness.  As Benjamin Franklin once said “An ounce of prevention is worth a pound of cure”.

3. Start testing your personnel on security procedures. One of the biggest security risks for any organization is staff mistakes and accidents that compromise credentials or data. Security awareness training makes an important difference. And if you test your personnel on security after the training, you verify that people understand the training and you identify areas of weakness. This makes it more likely that your personnel become more aware and follow better security practices. If you have access to an online assessment tool like Questionmark, it’s very, very easy to do.

Photo of doctor stethoscope on computer keyboard4. Review some of your key vendors. A risk for most organizations is weaknesses in suppliers or subcontractors that have access to your data. Ask suppliers to share information on their technical and organizational measures for security and what they are doing to ensure that your data is not breached. Any reputable organization will be willing and able to provide this under NDA. See 24 midsummer questions to ask your assessment software provider on this blog for some of the questions you can ask.

 

5. Conduct a restore test from backups. How do you know your backups work? Over the years, I’ve come across a few organizations and teams who’ve lost their data because their backups didn’t work. The only way to be sure is to test restoring it from backup and check data is there. If you don’t already run restore tests, organize a restore test in 2017 (ideally once a quarter, but once is better than not at all). You shouldn’t need to do this if you use a cloud service like Questionmark OnDemand as the vendor should do it for you.

6. Run a pilot for online proctoring. Microsoft do it. SAP do it. Why shouldn’t you do it? If you run a certification program that uses physical test centers, consider whether online proctoring might work for you. Not only will it reduce the risk of collusion with proctors helping candidates cheat, but it will also be a huge boon to your candidates who will no longer need to travel to test centers.

TheCadetHonorCodeMonument7. Put in place a code of conduct for your participants. This is a simple thing to do and can make a big difference in reducing cheating by encouraging test-takers to stay honest.  See Candidate Agreements: Establishing honor codes for test takers and What is the best way to reduce cheating? on this blog for tips on how and why to do this. If you are looking for inspiration, at famous code of conduct is that of the U.S. Army West Point Military Academy which simply says: “A cadet will not lie, cheat, steal, or tolerate those who do.” Of course you need to communicate and get buy-in for your code of conduct, but if you do, it can be very effective.

Many of you will already be doing all of these things, but if you’re not, I hope one or more of these resolutions help you improve your assessment security in 2017.

And here’s a bonus New Year’s resolution to consider. Questionmark Information Security Officer David Hunt and I are giving a session on Staying Ahead of Evolving Security Threats at the Questionmark conference in March in Santa Fe. Make a New Year’s resolution to come to the conference, and learn about security and assessment!

Online Proctoring: FAQs

John Kleeman HeadshotPosted by John Kleeman

Online proctoring was a hot-button topic at Questionmark’s annual Users Conference. And though we’ve discussed the pros and cons in this blog and even offered an infographic highlighting online versus test-center proctoring, many interesting questions arose during the Ensuring Exam Integrity with Online Proctoring  session I presented with Steve Lay at Questionmark Conference 2016.

I’ve compiled a few of those questions and offered answers to them. For context and additional information, make sure to check out a shortened version of our presentation. If you have any questions you’d like to add to the list, comment below!

What control does the online proctor have on the exam?

With Questionmark solutions, the online proctor can:

  • Converse with the participant
  • Pause and resume the exam
  • Give extra time if needed
  • Terminate the exam

What does an online proctor do if he/she suspects cheating?

Usually the proctor will terminate the exam and file a report to the exam sponsor.

What happens if the exam is interrupted, e.g. by someone coming in to the room?

This depends on your security protocols. Some organizations may decide  to terminate the exam and require another attempt. In some cases, if it seems an honest mistake, the organization may decide that the proctor can use discretion to permit the exam to continue.

Which is more secure, online or face-to-face proctoring?online proctoring

On balance, they are about equally secure.

Unfortunately there has been a lot of corruption with face-to-face proctoring, and online proctoring makes it much harder for participant and proctor to collude as there is no direct contact, and all communication can be logged.

But if the proctors are honest, it is easier to detect cheating aids in a face-to-face environment than via a video link.

What kind of exams is online proctoring good for?

Online proctoring works well for exams where:

  • The stakes are high and so you need the security of a proctor
  • Participants are in many different places, making travel to test centers costly
  • Participants are computer literate – have and know how to use their own PCs
  • Exams take 2-3 hours or less

If your technology or subject area changes frequently, then online proctoring is particularly good because you can easily give more frequent exams, without requiring candidates to travel.

What kind of exams is online proctoring less good for?

Online proctoring is less appropriate for exams where:

  • Exams are long and participants needs breaks
  • Exams where participants are local and it’s easy to get them into one place to take the exam
  • Participants do not have access to their own PC and/or are not computer literate

How do you prepare for online proctoring?

Here are some preparation tasks:

  • Brief and communicate with your participants about online proctoring
  • Define clearly the computer requirements for participants
  • Agree what happens in the event of incidents – e.g. suspected cheating, exam interruptions
  • Agree what ID is acceptable for participants and whether ID information is going to be stored
  • Make a candidate agreement or honor code which sets out what you expect from people to encourage them to take the exam fairly

I hope these Q&A and the linked presentation are interesting. You can find out more about Questionmark’s online proctoring solution here.