How many errors can you spot in this survey question?

John KleemanPosted by John Kleeman

Tests and surveys are very different. In a test, you look to measure participant knowledge or skill; you know what answer you are looking for, and generally participants are motivated to answer well. In a survey, you look to measure participant attitude or recollection; you don’t know what answer you are looking for, and participants may be disinterested.

Writing good surveys is an important skill. If you’re interested in how to write good surveys of opinion and attitude in training, learning, compliance, certification, based on research evidence, you might be interested in a webinar I gave titled, “Designing Effective Surveys.” Click HERE for the webinar recording and slides.

In the meantime, here’s a sample survey question. How many errors can you spot in the question?

The material and presentation qualty at Questionmark webinars is always excellent. Strongly Agree Agree Slightly agree Neither agree nor disagree Disagree Strongly disagree

There are quite a few errors. Try to count the errors before you look at my explanation below!!

I count seven errors:

  1. I am sure you got the mis-spelling of “quality”. If you mis-spell something in a survey question, it indicates to the participant that you haven’t taken time and trouble writing your survey, so there is little incentive for them to spend time and trouble answering.
  2. It’s not usually sensible to use the word “always” in a survey question. Some participants make take the statement literally, and it’s much more likely that webinars are usually excellent than that every single one is excellent.
  3. The question is double-barreled. It’s asking about material AND presentation quality. They might be different. This really should be two questions to get a consistent answer.
  4. The “Agree” in “Strongly Agree” is capitalized but not in other places, e.g. “Slightly agree”. Capitalization should be equal in every part of the scale.

You can see these four errors highlighted below.

Red marking corresponding to four errors above

Is that all the errors? I count three more, making a total of seven:

  1. The scale should be balanced. Why is there a “Slightly agree” and not a “Slightly disagree”?
  2. This is a leading or “loaded” question, not a neutral one, it encourages you to a positive answer. If you genuinely want to get people’s opinion in a survey question, you need to ask it without encouraging the participant to answer a particular way.
  3. Lastly, any agree/disagree question has acquiescence bias. Research evidence suggests that some participants are more likely to agree when answering survey questions. Particularly those who are more junior or less educated who may tend to think that what is asked of them might be true. It would be better to word this question to ask people to rate the webinars rather than agree with a statement about them.

Did you get all of these? I hope you enjoyed this little exercise. If you did, I explain more about this and good survey practice in our Designing Effective Surveys webinar, click HERE for the webinar recording and slides.

Beyond Recall : Taking Competency Assessments to the Next Level

A pyramid showing create evaluate analyze apply understand remember / recall

John KleemanPosted by John Kleeman

A lot of assessments focus on testing knowledge or facts. Questions that ask for recall of facts do have some value. They check someone’s knowledge and they help reduce the forgetting curve for new knowledge learned.

But for most jobs, knowledge is only a small part of the job requirements. As well as remembering or recalling information, people need to understand, apply, analyze, evaluate and create as shown in Bloom’s revised taxonomy right. Most real world jobs require many levels of the taxonomy, and if your assessments focus only on recalling knowledge, they may well not test job competence validly.

Evaluating includes exercising judgement, and using judgement is a critical factor in competence required in a lot of job roles. But a lot of assessments don’t assess judgement, and this webinar will explain how you can do this.

There are many approaches to creating assessments that do more than test recall, including:

  • You can write objective questions which test understanding and application of knowledge, or analysis of situations. For example you can present questions within real-life scenarios which require understanding a real-life situation and working out how to apply knowledge and skills to answer it. It’s sometimes useful to use media such as videos to also make the question closer to the performance environment.
  • You can use observational assessments, which allow an observer to watch someone perform a task and grade their performance. This allows assessment of practical skills as well as higher level cognitive ones.
  • You can use simulations which assess performance within a controlled environment closer to the real performance environment
  • You can set up role-playing assessments, which are useful for customer service or other skills which need interpersonal skills
  • You can assess people’s actual job performance, using 360 degree assessments or performance appraisal.

In our webinar, we will give an overview of these methods but will focus on a method which has always been used in pre-employment but which is increasingly being used in post-hire training, certification and compliance testing. This method is Situational Judgement Assessments – which are questions carefully written to assess someone’s ability to exercise judgement within the domain of their job role.

It’s not just CEOs who need to exercise judgment and make decisions, almost every job requires an element of judgement. Many costly errors in organizations are caused by a failure of judgement. Even if people have appropriate skill, experience and knowledge, they need to use judgement to apply it successfully, otherwise failures occur or successful outcomes are missed.

Situational Judgment Assessments (SJAs) present a dilemma to the participant (using text or video)  and ask them to choose options in response. The dilemma needs to be one that is relevant to the job, i.e. one where using judgement is clearly linked to a needed domain of knowledge, skill or competency in the job role. And the scoring needs to be based on subject matter experts alignment that the judgement is the correct one to make.

Context is defined (text or video); Dilemma that needs judgment; The participant chooses from options; A score or evaluation is made

Situational Judgement Assessments can be a valid and reliable way of measuring judgement and can be presented in a standalone assessment or combined with other kinds of questions. If you’re interested in learning more, check out our webinar titled “Beyond Recall: Taking Competency Assessments to the Next Level.” You can download the webinar recording and slides HERE.

How is the SAP Global Certification program going? A re-interview with SAP’s manager of global certification, part 2.

Posted by Zainab Fayaz

This is the second part of the two-part interview between, John Kleeman, Founder and Executive Director at Questionmark and Ralf Kirchgaessner, Manager of the SAP Global Certification program. This is a continuation of the use of Questionmark software in their Certification in the Cloud program. You can read the first part here. In the second part of the interview, John asks questions about the business benefits of certification and what advice Ralf has for other organizations.

John: What are the business benefits to SAP of certification?

Ralf: There are many benefits to the SAP Global Certification. So, let’s begin from the individual learner’s perspective.

Firstly, earning the SAP Global Certification increases your personal value; not only does it drive personal development; which often leads to increased responsibilities and promotion within your organization, but it also showcases and proves that you stay current and update your skills to the latest releases. Additionally, since 2018, professionals can gain wider recognition through sharing their SAP Global Certification digital badges.

SAP Global Certification is of great value not only for individuals but also for consultancies in the SAP ecosystem. SAP Global Certifications provide a clear measure of a company’s organizational capabilities, which give a competitive advantage, especially if the company has certified professionals in new and innovative areas, like SAP C/4HANA Cloud.

John: What about the customers? What benefits are there for them?

Ralf: Indeed, the most important benefit is the value for our customers. If SAP can ensure that the consultancy eco-system is well enabled and certified, it helps reduce the total cost of ownership (TCO) and ensures successful implementation costs. And in the end, this is of course also important for SAP, as this helps to increase the adoption of our software and reduces implementation risks.

John: Tell me a bit more about the recently introduced digital badges for people who get certified that you just mentioned. How useful is that?

Ralf: The introduction of digital badges for SAP Global Certification has been an absolute success! Making your workforce visible on the market is important and by sharing the digital badge proves that the workforce is currently in their knowledge. If on LinkedIn, you search for ‘certified SAP consultants’, you would find thousands of shared badges. Digital badge claim rates beyond industry standards show that people waited with much anticipation to share their achievements digitally.

We are constantly looking for ways to improve our services and with the help of Questionmark, going forward we will be able to issue badges, even faster. In the near future, once candidates have passed their SAP Global Certification exam this will trigger the issuing of badges in “real-time”!

We have reached our ultimate goal and an overall mission of our certification programme if customers ask consultants for their digital badges to show their SAP Global Certification status.

John: There seems a slow move across the community from test centers to online proctoring. I know that for SAP, you deliver some exams in your offices but most in the cloud with online proctoring. How do you see this changing in the industry in general? Will all IT exams be done by online proctoring one day soon?

Ralf: SAP very much uses the model of taking exams wherever and whenever it is most convenient. Nevertheless, we use one harmonized infrastructure, for all our exams and these can be taken at our offices, in classrooms or in the cloud.

I think much of this evolves from the changing landscape in learning behaviors and offerings. In terms of the advantages of using test centers and online proctoring; there is a legitimate reason for test centres to exist; as there are groups of people who will still want to learn together – in one place at one time. However, as the shift moves towards a rise in remote learning, both synchronous (live virtual classrooms) and asynchronous, which are supported by social and peer learning via online learning rooms, then of course, online proctoring will become more popular.

John: What advice would you give to other high-tech companies who are thinking of setting up or improving their certification program?

Ralf: Two things instantly come to mind – online proctoring and digital badging. Certification programs that do not use online proctoring and digital badging should urgently consider improving their program as the benefits of implementing both features are tremendous.

More on certification
Interested in learning more about certification programs?  Find out how you can build your own certification program in 10-easy steps.

 

How is the SAP Global Certification program going? A re-interview with SAP’s manager of global certification, part 1.

Posted by Zainab Fayaz

Back in 2016, John Kleeman, Founder and Executive Director of Questionmark interviewed Ralf Kirchgaessner, Manager of SAP Global Certification program about their use of Questionmark software in their Certification in the Cloud program and about their move to online proctoring. You can see the interview on the Questionmark blog here. We also thought readers might be interested in an update, so here is a short interview between the two on how SAP are getting on three years later:

John: Could you give us an update on where you are with the Certification in the Cloud program?

Ralf: The uptake, adoption and increase of Certification in the Cloud is tremendous! Over the years we have seen a significant increase in the volume of candidates taking exams in the cloud; the numbers doubled from 2016 to 2017 and increased almost by 60% in 2018. This means more than 50% of SAP Global Certification exams are now done remotely!

John: Are all your SAP Global Certification exams now available online in the cloud?

Ralf: Nearly so. By mid-2019 we plan on having the complete portfolio of every SAP exam available on the cloud. This is great news for our learners who have invested in a Certification in the Cloud subscription. So, we then have Certification in the Cloud not only for SAP SuccessFactors and SAP Ariba, but for all products, including SAP C/4HANA.

John: How many different languages are your exams translated into?

Ralf: This depends on the portfolio. Some of our certifications are available in English and others, such as for SAP Business One are translated in up to 20 languages.

John: How are you dealing with the fast pace of change within SAP software in a certification context? How do you ensure certifications stay up to date when the software changes?

Ralf: This is of course a challenge. In previous years, it was the case of getting certified once every few years. However, now you must keep your skills up-to-date and stay current with quarterly release cycles of our SAP Cloud solutions. Also, for people who are first timers or newly enter the SAP eco-system; it is important that they are certified on the latest quarterly release.

To help overcome this challenge, we have developed an agile approach to updating our exams; we use the Questionmark platform for those who are new to the eco-system to help them getting certified initially. We also have a very good process in place and often use the same subject matter experts when it comes to keeping up to the speed of software changes.

For already certified professionals, another way to remain up to date is through our ‘Stay Current’ program. For some of our solutions, partners have to come back every 3 months to show that they are staying current. They do this in the form of taking a short “delta” knowledge assessment. For instance, for certified professionals of SAP SuccessFactors it is mandatory to stay current in order to get provisioning access to the software systems.

In 2018, SAP’s certification approach was acknowledged with the ITCC Innovation Award. Industry peers like from Microsoft, IBM and others recognized this great achievement with this award.

 

Q&A: Sue Martin and John Kleeman discuss steps to building a certification program

Posted by Zainab Fayaz

Certification programs are a vital way of recognizing knowledge, skills and professional expertise, but, during a time of digital transformation, how do you build a program that is sustainable and adaptable to the evolving needs of your organization, stakeholders and the market?

Questionmark Founder and Executive Director, John Kleeman and Sue Martin, certification expert and Business Transformation Consultant presented a webinar on how to build a certification program (you can view the webinar HERE). Before the webinar, we sat down with our experts to gain some insight on what they’ll be covering during the session.

Tell us a bit about what you’ll be covering during the webinar:

Sue: During the webinar, we’ll be covering a range of things; from the conceptual steps of building a certification program to the many projects that have evolved from these and the importance of outlining key steps from the very beginning of the process for creating a comprehensive and cohesive certification program.

We will also talk about the value certification program, can add to an organization, not only in the short-haul but also for many years to come. It is important to remember, “why” and “what” you are trying to achieve, and this webinar will provide detail on how the alignment of strategic goals and communication with stakeholders contributes to the success of an adaptable certification program.

John: We’ll be discussing a range of things during the webinar, but here are the ten easy steps that we’ll be describing:

  1. Business goals
  2. Scope
  3. Security
  4. Vendor evaluation
  5. Blueprint and test design
  6. Test development
  7. Pilot
  8. Communications
  9. Delivery
  10. Reporting and monitoring

What influenced the selection of these 10-steps you have identified in building a certification program?

John:  Sue and I sat down to plan the webinar when we were together at the OEB conference in Berlin in December. Although we wanted to cover a bit some of the obvious things like test design and development, we wanted to make sure people think first about the preparation and planning, for example getting organization buy-in and working out how to market and communicate the program to stakeholders. So we’ll be focusing on what you need to do to make a successful program, and that will drive everything you do

Although you’ll be covering the key steps for building a certification program during the webinar, can you advise on three key steps you find to be the most important during the process:

Sue:
1. Planning:
The emphasis of the program’s work should be at the start, in the planning phase – especially in order to build a flexible program which will adapt to the needs of your audience and stakeholders as their needs change over time. In all of the individual project components, whether it be test creation, vendor evaluation or communications rollout, for example, design and plan for the end goal. For example, when it comes to creating an exam, you plan for it right at the start of the project – you hit the ground running! It is not all about item writing, but also the development of the project from the beginning and if you don’t plan; this can lead to the lack of validity in the exam program and inconsistency over time

2. Practical tips and tricks for approaching various elements of your program development: It is important to set out the target audience; identify their learning journey and how they learn – in knowing this, can you go forward and build a certification program that can become integrated and aligns with the learning process

3. Scope: This is very important; setting the scope is a priority. Of course, in the greater scheme of things; you’ll have a mission statement, which provides you with a strategic vision, but when it comes to the finer detail and knowing what countries to enter, the pricing structure or knowing whether to offer remote proctoring; always keep in mind three things: the value contribution, the stakeholders and ask yourselves “yes, but why?”; as this will help align with organizational objectives.

What can attendees take away from the webinar you’ll present?  

Sue: Those attending will learn the value and importance of planning and questioning everything from the start of the process. We’ll share advice on the importance of having a value statement for every part of the process and making sure you know that a certification program is what you are looking for. By attending you can walk away with knowing the operational and strategic steps you must go through in order to build a program that is sustainable; think of it as a checklist!

John: If you’re starting a new certification program, I think this webinar will help guide you and help you create it more easily and more effectively. And if you already have a certification program and want to improve it, you’ll probably be doing a lot of what we suggest already but I hope they’ll be something for everyone to take away and learn.

Want to know more?

If you’re interested in learning more about the steps to building a certification program that meets the needs of your organization and stakeholders, check out John and Sue’s webinar session, Building a Certification Program in 10 easy steps.

A little bit more about our two experts:

John Kleeman is Executive Director and Founder of Questionmark. He has a first-class degree from Trinity College, Cambridge, and is a Chartered Engineer and a Certified Information Privacy Professional/Europe (CIPP/E). John wrote the first version of the Questionmark assessment software system and then founded Questionmark in 1988 to market, develop and support it. John has been heavily involved in assessment software for 30 years and has also participated in several standards initiatives including IMS QTI, ISO 23988 and ISO 10667. John was recently elected to the Association of Test Publishers (ATP) Board of Directors.

Sue Martin is a trusted advisor to companies and institutions across Europe in the area of workforce credentialing, learning strategies and certification. Her career prior to consulting included a role as Senior Global Certification Director for SAP and several regional and global management roles in the testing industry. She has also held several positions within industry institutions, such as the Chair of the European Association of Test Publishers and is currently a member of the Learning & Development Committee at BCS (British Computer Society).

Q&A: Microlearning and the Role of Measurement in Learning and Development at Progressive

 

Posted by Kristin Bernor

Chris Gilbert is a Senior Instructional Designer for Progressive Insurance, one of the largest providers of insurance in the United States. During his case study presentation at the 2019 Questionmark Conference in San Diego taking place from February 26 – March 1, he will talk about Using Questionmark to Build Microlearning for Photo Estimators. Progressive photo estimators use videos and photographs to identify damage and write estimates for necessary repairs.

This session will explore Progressive’s use of microlearning modules and the process they use to develop them.

I asked him recently about their case study:

Tell us about Progressive and your use of assessments:

At Progressive, we seek to make informed, data-driven decisions. We also strive to prepare and develop our people through effective, targeted learning solutions. In our learning orgs, assessments are one mechanism for gathering data we use to make a variety of decisions, including:

  • Identifying aspects and features of learning experiences that resonate within our target audiences so we can implement them in more of our deliverables
  • Pinpointing opportunities to improve learning experiences for our target audiences

Over the past few years, we’ve had a renewed focus on the importance of learning measurement and have established and implemented standards and tools for performing Level 1 and Level 2 measurement across and within all of our learning organizations. We’re currently working on Level 3 measurement to be able to measure and communicate the on-the-job impact of our learning experiences more consistently.

What do you mean by microlearning and why is it important to Progressive?

Microlearning is skill-based learning delivered in small “bite-sized” pieces. Microlearning can be developed in a variety of formats including videos, games, scenarios, and several others. Depending on the situation and needs of the organization and learners, microlearning can be delivered standalone, or as a supplement other learning experiences like in-person or virtual classroom courses.

Progressive is interested in adding microlearning into our learning deliverable portfolio for a variety of reasons, including:

  • Faster development times can improve our ability to deliver just-in-time learning solutions at the speed of modern business and change
  • Tightly-focused, skill-based topics and practice directly support on-the-job application
  • Today’s corporate learners seek quick-hit learning that gives them practical tools they need to succeed
  • Delivered in conjunction with other learning experiences, microlearning can help learners overcome the forgetting curve

What role does Questionmark play in ensuring that microlearning is successful?

One of the primary reasons, we decided to use Questionmark for our microlearning pilot project is that the data the system captures and it’s reporting capabilities allow us to provide the business with insights into several aspects of the learners’ performance in the modules. In turn, these insights will help the business make informed decisions.

What else about your session would you like to share?

Besides sharing the story of our first foray into microlearning, I’m planning to discuss some of the learnings we had related to question-type capabilities that we hadn’t previously explored.

Who would benefit most from attending this session and why?

a. Anyone interested in using Questionmark beyond its traditional use because the way we’re using it is a bit unconventional

b. Anyone interested in adding microlearning to their learning deliverable portfolio because Questionmark may provide a way for them to develop, deliver, and report the results

c. Anyone interested in extending the functionality of Questionmark question types to meet a business need because I’ll dig into some of the challenges, realizations, and learnings I experienced from having to extend a few of the question types in the pilot project

What are you especially looking forward to at this year’s Questionmark conference?

Meeting and networking with other Questionmark users, especially those who are passionate about the role of measurement in learning and development, and gaining more insight into how others are using system features and functionality in their organizations

Thank you Chris for taking time out of your busy schedule to discuss your session with us!

***

If you have not already done so, you still have a chance to attend this important learning event. Click here to register.