How is the SAP Global Certification program going? A re-interview with SAP’s manager of global certification, part 1.

Posted by Zainab Fayaz

Back in 2016, John Kleeman, Founder and Executive Director of Questionmark interviewed Ralf Kirchgaessner, Manager of SAP Global Certification program about their use of Questionmark software in their Certification in the Cloud program and about their move to online proctoring. You can see the interview on the Questionmark blog here. We also thought readers might be interested in an update, so here is a short interview between the two on how SAP are getting on three years later:

John: Could you give us an update on where you are with the Certification in the Cloud program?

Ralf: The uptake, adoption and increase of Certification in the Cloud is tremendous! Over the years we have seen a significant increase in the volume of candidates taking exams in the cloud; the numbers doubled from 2016 to 2017 and increased almost by 60% in 2018. This means more than 50% of SAP Global Certification exams are now done remotely!

John: Are all your SAP Global Certification exams now available online in the cloud?

Ralf: Nearly so. By mid-2019 we plan on having the complete portfolio of every SAP exam available on the cloud. This is great news for our learners who have invested in a Certification in the Cloud subscription. So, we then have Certification in the Cloud not only for SAP SuccessFactors and SAP Ariba, but for all products, including SAP C/4HANA.

John: How many different languages are your exams translated into?

Ralf: This depends on the portfolio. Some of our certifications are available in English and others, such as for SAP Business One are translated in up to 20 languages.

John: How are you dealing with the fast pace of change within SAP software in a certification context? How do you ensure certifications stay up to date when the software changes?

Ralf: This is of course a challenge. In previous years, it was the case of getting certified once every few years. However, now you must keep your skills up-to-date and stay current with quarterly release cycles of our SAP Cloud solutions. Also, for people who are first timers or newly enter the SAP eco-system; it is important that they are certified on the latest quarterly release.

To help overcome this challenge, we have developed an agile approach to updating our exams; we use the Questionmark platform for those who are new to the eco-system to help them getting certified initially. We also have a very good process in place and often use the same subject matter experts when it comes to keeping up to the speed of software changes.

For already certified professionals, another way to remain up to date is through our ‘Stay Current’ program. For some of our solutions, partners have to come back every 3 months to show that they are staying current. They do this in the form of taking a short “delta” knowledge assessment. For instance, for certified professionals of SAP SuccessFactors it is mandatory to stay current in order to get provisioning access to the software systems.

In 2018, SAP’s certification approach was acknowledged with the ITCC Innovation Award. Industry peers like from Microsoft, IBM and others recognized this great achievement with this award.

 

Q&A: Sue Martin and John Kleeman discuss steps to building a certification program

Posted by Zainab Fayaz

Certification programs are a vital way of recognizing knowledge, skills and professional expertise, but, during a time of digital transformation, how do you build a program that is sustainable and adaptable to the evolving needs of your organization, stakeholders and the market?

Questionmark Founder and Executive Director, John Kleeman and Sue Martin, certification expert and Business Transformation Consultant will present the upcoming webinar Building a Certification Program in 10 easy stepsat 4pm (GMT) / 11am Eastern / 5pm CET on 12 February 2019. However, before then, we wanted to find out a little bit more about what they’ll be covering during the session, so we’ve interviewed the two experts to gain insight!

Tell us a bit about what you’ll be covering during the webinar:

Sue: During the webinar, we’ll be covering a range of things; from the conceptual steps of building a certification program to the many projects that have evolved from these and the importance of outlining key steps from the very beginning of the process for creating a comprehensive and cohesive certification program.

We will also talk about the value certification program, can add to an organization, not only in the short-haul but also for many years to come. It is important to remember, “why” and “what” you are trying to achieve, and this webinar will provide detail on how the alignment of strategic goals and communication with stakeholders contributes to the success of an adaptable certification program.

John: We’ll be discussing a range of things during the webinar, but here are the ten easy steps that we’ll be describing:

  1. Business goals
  2. Scope
  3. Security
  4. Vendor evaluation
  5. Blueprint and test design
  6. Test development
  7. Pilot
  8. Communications
  9. Delivery
  10. Reporting and monitoring

What influenced the selection of these 10-steps you have identified in building a certification program?

John:  Sue and I sat down to plan the webinar when we were together at the OEB conference in Berlin in December. Although we wanted to cover a bit some of the obvious things like test design and development, we wanted to make sure people think first about the preparation and planning, for example getting organization buy-in and working out how to market and communicate the program to stakeholders. So we’ll be focusing on what you need to do to make a successful program, and that will drive everything you do

Although you’ll be covering the key steps for building a certification program during the webinar, can you advise on three key steps you find to be the most important during the process:

Sue:
1. Planning:
The emphasis of the program’s work should be at the start, in the planning phase – especially in order to build a flexible program which will adapt to the needs of your audience and stakeholders as their needs change over time. In all of the individual project components, whether it be test creation, vendor evaluation or communications rollout, for example, design and plan for the end goal. For example, when it comes to creating an exam, you plan for it right at the start of the project – you hit the ground running! It is not all about item writing, but also the development of the project from the beginning and if you don’t plan; this can lead to the lack of validity in the exam program and inconsistency over time

2. Practical tips and tricks for approaching various elements of your program development: It is important to set out the target audience; identify their learning journey and how they learn – in knowing this, can you go forward and build a certification program that can become integrated and aligns with the learning process

3. Scope: This is very important; setting the scope is a priority. Of course, in the greater scheme of things; you’ll have a mission statement, which provides you with a strategic vision, but when it comes to the finer detail and knowing what countries to enter, the pricing structure or knowing whether to offer remote proctoring; always keep in mind three things: the value contribution, the stakeholders and ask yourselves “yes, but why?”; as this will help align with organizational objectives.

What can attendees take away from the webinar you’ll present?  

Sue: Those attending will learn the value and importance of planning and questioning everything from the start of the process. We’ll share advice on the importance of having a value statement for every part of the process and making sure you know that a certification program is what you are looking for. By attending you can walk away with knowing the operational and strategic steps you must go through in order to build a program that is sustainable; think of it as a checklist!

John: If you’re starting a new certification program, I think this webinar will help guide you and help you create it more easily and more effectively. And if you already have a certification program and want to improve it, you’ll probably be doing a lot of what we suggest already but I hope they’ll be something for everyone to take away and learn.

Want to know more?

If you’re interested in learning more about the steps to building a certification program that meets the needs of your organization and stakeholders; then join John and Sue deliver a webinar session dedicated to Building a Certification Program in 10 easy steps on 12 February 2019 at 4pm (GMT) / 11am Eastern / 5pm CET.

You can register here.

A little bit more about our two experts:

John Kleeman is Executive Director and Founder of Questionmark. He has a first-class degree from Trinity College, Cambridge, and is a Chartered Engineer and a Certified Information Privacy Professional/Europe (CIPP/E). John wrote the first version of the Questionmark assessment software system and then founded Questionmark in 1988 to market, develop and support it. John has been heavily involved in assessment software for 30 years and has also participated in several standards initiatives including IMS QTI, ISO 23988 and ISO 10667. John was recently elected to the Association of Test Publishers (ATP) Board of Directors.

Sue Martin is a trusted advisor to companies and institutions across Europe in the area of workforce credentialing, learning strategies and certification. Her career prior to consulting included a role as Senior Global Certification Director for SAP and several regional and global management roles in the testing industry. She has also held several positions within industry institutions, such as the Chair of the European Association of Test Publishers and is currently a member of the Learning & Development Committee at BCS (British Computer Society).

 

How to manage compliance in a highly regulated world? [30-minute webinar]

Posted by Chloe Mendonca

If your industry demands compliance, then your people need compliance training and certifications. Learning and training records are almost always reviewed during regulatory audits to check that employees have received the required training and their competencies or certifications are up to date and valid. The regular assessment of employee knowledge and competencies ensures you’re always ready for an audit or if something goes wrong.

Did you know…


Source: Brandon Hall Group

Perhaps you already understand the value assessments can bring, but need to convince your management team. Or perhaps you’d like a better view on how to use assessments most effectively to ensure compliance. You’re not alone.

Join us for a 30-minute webinar on Thursday March 9, 2017 to:

  • discuss the critical role assessments play in compliance learning
  • explore the benefits of using assessments before, during and after training
  • find out 7 ways assessments fortify compliance
  • get best practice tips for ensuring valid and reliable assessments

We also have several other webinars you may be interested in:

  • How to write high-quality test items – March 4, 2017

If you’d like best practice tips to improve your test items and ensure they produce fair, valid and reliable results then sign up for this 30-minute webinar.

  • Introduction to Questionmark’s Assessment Management System – Various dates and times

Get an overview of Questionmark’s features and functions in this live demo. We will look at the basics of authoring, delivering and reporting on surveys, quizzes, tests and exams.

Online or test center proctoring: Which is best?

John Kleeman HeadshotPosted by John Kleeman

A new way of proctoring certification exams is rapidly gaining traction. This article compares and contrasts the old with the new.

Many high-tech companies offer certification exams for consultants, users and implementers. Such exams often require candidates to travel to a bricks-and-mortar test center where proctors (or invigilators) supervise the process.

Now, however, online proctoring is becoming prevalent: each candidate takes the exams at his or her home or office, with a proctor observing via video camera over the Internet. Two of the world’s largest software companies, SAP and Microsoft, offer online proctoring for their certification programs, and many other companies are looking to follow suit. This article explains some of the pros and cons of the two approaches.

workplace_addFactors for choosing online proctoring

  • Reduced travel time.  Candidates can take an exam without wasting time traveling to a test center. This is an important saving for their employers – often the test sponsor’s customers.
  • Convenient scheduling. A candidate can choose a convenient time, for example after the kids have gone to bed or when work pressures are lowest. Usually one needs to book in advance to attend a test center, but it’s often possible to schedule an online proctor at short notice.
  • Fairness. With an exam at a test center, some people will have had a short journey and others a longer one. Some might have experienced a traffic jam or other hassle getting there. This gives an advantage to those who happen to live closer, as they will have less anxiety. An online experience reduces the variability of the exam experience.
  • Accessibility. Candidates take online proctored exams on their own computers, using their normal accessibility aids such as screen readers or special input devices, whereas these require setup at a test center. Some test centers only provide their own (often limited) tools for providing accommodations, so candidates are working with unfamiliar tool sets. This places them at a disadvantage. Also, for people with certain disabilities, travel is a major inconvenience.
  •  Keeping certifications up to date. If candidates have to travel to a test enter, a test sponsor can’t realistically require an exam to be taken more than once every few years. But in today’s world, products and job skills change very quickly, so certification risks being out of date. The availability of online  proctoring allows update exams (assessing candidates on what has changed since their last exam) to be taken as products change, which makes the programme more valid.
  • Greater authenticity. The more authentic assessments are, the more they measure actual performance. See Will Thalheimer’s excellent paper on measuring learning results for more on this. Assessing someone in their work environment with online proctoring is more authentic and so will likely measure performance better than putting them in a test center.

office-buildingFactors for choosing test center proctoring

  • Standardized computers. While online proctoring requires the candidate to have an appropriate computer, internet connection and webcam that they know how to use, test centers provide a computer that is already set up. For most certification programmes, it’s fair and reasonable that candidates use their own computers (often called BYOD – Bring Your Own Device). But for some programmes, this might be less fair. For example, in professions where IT literacy is not required, it might not be fair to expect people to have access to a PC with webcam that they know how to use.
  • Very long exams.  In online-proctored exams, the candidate is usually forbidden from taking a break for security reasons.  Most exams can be taken in one sitting, but if exams are longer than three hours, a test center makes sense.
  • Regulation. Some regulators or government authorities may require delivery of an exam with a physically present proctor at a test center.
  • Geographical convenience.  In some cases, test centers may be close at hand. For example, a university might have all its candidates already present, or, for some test sponsors, candidates may all live in metropolitan areas close to test centers.

checkOther factors to consider:

  • Language. In theory, a candidate could schedule an online proctor in his or her own language, though in practice many programs only offer English-speaking proctors. A test center may well not have proctors who can speak different languages, but typically will speak the local language.
  • Security. You might think that the security is stronger in a test center than with online proctoring. However, over the years there have been many incidents where face-to-face proctors have coached candidates. Online proctoring also makes it feasible to administer exams more frequently, which helps security by making impersonation harder. This is a big subject, and I’ll follow up with a blog post about security.

I’d welcome your thoughts on any other factors for and against online proctoring.

How many test or exam retakes should you allow? Part 2

John Kleeman HeadshotPosted by John Kleeman

In my last post, I offered some ideas about what to consider when determining your retake policy regarding a certification assessment measuring competence and mastery. Some of the issues to balance are test security, fairness, a delay between retakes and the impact of retakes on test preparation. In this conclusion to the post, I’ll share what a few other organizations do and how you might approach deciding the number of retakes to allow.

Here is how a few respected certification programmes manage retakes

SAP have the following rules in their certification programme:

No candidate may participate in the same examination for the same release more than three times. A candidate who has failed at an examination three times for a release may not attempt that examination again until the next release. 

Microsoft allow up to 5 attempts in a 12-month period and then impose a 12-month waiting period. They also have gaps of several days between retakes, with the number of days increasing for subsequent retakes.

The US financial regulator FINRA requires a waiting time of 30 days between exams, but if you fail an exam three or more times in succession, you must wait 6 months before taking it again.

What’s the right answer for you?

The right answer depends on your circumstances. Many programmes allow retakes but have rules in place to limit the delivery rate of the assessment in order to limit content exposure.

1. You should communicate your retake policy to participants and to stakeholders who see the results of the assessments.

2. If you release scores, you also need to decide whether you will have a policy  as to whether scores for all attempts are released, or (as many organizations do) only for the successful attempt. Section 11.2 of the the Standards for Educational and Psychological Testing states

“Test users or the sponsoring agency should explain to test takers their opportunities, if any, to retake an examination; users should also indicate whether the earlier as well as later scores will be reported to those entitled to receive score reports.”

3. You should not allow people to retake a test they have passed.

4. You should consider requiring a period of time to elapse before someone retakes an exam if they fail. This allows time for them to update their learning. You can easily set this up when scheduling within Questionmark software, for example the dialog below gives a 7-day gap.

Limit days between retakes

5. Unless special circumstances apply, you will usually want to allow at least one retake and probably at least two retakes.

6. You may want to consider some intervention or stop procedure after a certain number of failed attempts. A common number I’ve heard anecdotally is three attempts, but it will depend on each assessment program’s own individual factors and use cases.  If this is an internal compliance exam, you might want to organize some remedial training or job review. If this is a public exam, you might want to ensure a longer time period to allow reflection and re-learning.

Please feel free to comment below if you have alternative thoughts on the number of retakes to allow.

Using Questionmark to conduct a Performance Based Certification

John Kleeman HeadshotHow do you measure people’s practical skills, for example their ability to conduct electrical or electronic work? Is it possible to have someone use test equipment such as a hardware device or simulator and have this controlled by Questionmark technology?

Dave Fricton and Carolyn Parise The answer to this question is “yes”. I’d like to share an interesting story by Questionmark customer SpaceTEC, whose very inventive approach is making this happen.

SpaceTEC is the National Science Foundation, National Resource Center for Aerospace Technical Education. I’m grateful to DSpaceTec logoave Fricton and Carolyn Parise of SpaceTEC (pictured right) for presenting about this at the Questionmark Users Conference in San Antonio, Texas and for helping me write this blog article.

SpaceTEC, under its sister organization CertTEC, created an electronics certification that is offered to the electrical/electronics industry. Staff members there deliver knowledge based assessments with Questionmark and also deliver a practical exam where someone has to actually perform some electrical work, for instance finding a fault or make measurements. To do this, they use electrical test consoles and proprietary card sets like those shown below.

Model 130E Test console from NIDA                                 Electronic console

Traditionally the practical exam has been delivered manually by an examiner working with the candidate on the test equipment. But this is costly and difficult to organize nationwide as exams take 3 to 4 hours each and examiners need specialized training.

The innovation by SpaceTEC is that they have inserted HTML code inside Questionmark questions to control the test equipment. They drive the test equipment from within Questionmark software, making it no longer necessary for a trained examiner to run the practical test. They still have a proctor in the room to check the integrity of the process, but this is much easier to organize.

Here is a simple example of the kind of question they deliver. The candidate clicks on the Insert Fault button and this sets up the console with the appropriate configuration. Then the candidate measures the resistance on the console and types in their answer to Questionmark, which records and scores the question.

What is the resistance measurement in Kohms between TP1 and TP2?

If you want to know what happens behind the scenes, it’s very simple. The console (from NIDA Corporation) has an ActiveX driver which can be called by HTML code. It is necessary to install the driver on the candidate PC and then plug the console into the PC, but these are simple tasks. The call to the ActiveX driver is then encapsulated inside the Questionmark question wording.

Screenshot of HTML used

This is an example of Questionmark’s Open Assessment Platform, where you can connect Questionmark into other systems to get the solution you need. To quote Dave Fricton in his conference session: “The beauty of using Questionmark is you can do it all yourself”

Do you deliver any practical or performance exams using test equipment? If so, you might be able to follow the same route that SpaceTEC have gone, and link these up to Questionmark for easier administration. SpaceTEC are showing how performance and practical tests can be run in Questionmark, as well as knowledge assessments.