How Topic Feedback can give Compliance Assessments Business Value

Posted by John Kleeman

If you need to prove compliance with regulatory requirements, should your training and assessments focus on compliance needs? Or should you train and assess to improve skills that will impact your business primarily, and meet compliance needs as well?

I recently interviewed Frederick Stroebel and Mark Julius from a large South African financial services company, Sanlam Personal Finance, for the SAP blog. Sanlam have used Questionmark Perception for more than a decade and combine it with SAP HR and Learning software. You can see the full interview on the SAP site. Their view was that compliance and business-related needs must be combined:

“We deliver assessments both for compliance and e-learning. It’s a combination of business requirements and legislation. We predominantly started it off thinking that the purpose would be for business reasons, but as soon as the business realized the value for regulatory compliance, we received more and more requests for that purpose.”

One of the key ways in which Sanlam use the results of assessments to improve feedback is to use topic feedback, which identifies topics that may be weak points for the participant.

We set up our assessments so that at the end, the computer gives the participant a summary of the topics and what the score was per topic, so the participant can immediately see where they need further facilitation as well.

It is also valuable in providing feedback to the learner, where a facilitator sits with the learner. The facilitator can immediately determine from the coaching report where exactly the learner needs to go for re-training. We have done extremely well in terms of increasing our overall pass mark and per topic scores by using topic feedback.  For example, for brokers and advisers, there’s an initial assessment they do, and because questions are in different topics, once they’ve taken the assessment, the facilitator can immediately see which type of training that person must go on.

To illustrate, here is part of a Coaching Report that shows a participant has scored 80% in one topic (well above what is needed for competency) and 58% in another (slightly above what is needed).


Questionmark Perception coaching report

Topic feedback is a great way of getting value from assessments and I hope Sanlam’s experience and insight can help you.

How SAP creates educational and assessment content

Posted by John Kleeman

As I’ve mentioned in previous posts, Questionmark are partners of SAP, so I blog on the SAP Community Site as well as here on Questionmark’s blog.

Dr. Thilo Buchholz

In my SAP blog I’m featuring Q&A interviews with thought leaders in the field of assessment and learning within the SAP ecosystem. I spoke recently with Dr. Thilo Buchholz, who leads a team that handles production platform management and operations for SAP Education. Our interview, which covered many aspects of the systems that are used to create all of SAP educational content, was featured in the SAP Business Process Expert Community Newsletter.

Among the various subjects we covered was SAP’s move toward mobile. Here’s just a bit of that interchange.

John : As more SAP software runs on mobiles, do you see delivering learning content on mobiles will be more important?

Thilo : Definitely yes. SAP has a strategy of delivering business applications On Premise, On Demand and On Device. And as we deliver our software in these categories, our knowledge transfer, the actual performance support needs to be delivered via the same distribution channels. So we will be publishing educational content On Device if the software is delivered on Device. Actually our single source approach provides us with the flexibility to deliver educational content wherever it is needed.

Questionmark Perception could serve in this environment by flexibly supporting assessment scenarios. One example is to deliver a pre-assessment prior to a learning object; if the learner can answer the questions, then he or she can skip the learning object. Secondly in the learning process the learner can use self-assessments in order to check progress. And after the learner has passed the learning object it’s good to be able to have some control to see if he or she achieved the learning objective.  So overall, an assessment scenario is enriching e-learning content and helps the learner to steer themselves well, without a face-to-face trainer. Remember that in a classroom environment the instructor frequently checks on the learning progress of participants and adjusts approach based on this result of his/her “assessment”.

To see the rest of the interview, including questions on how SAP sees technology influencing and changing the future production of training materials, click here.

How SAP ensure their questions are good quality

Posted by John Kleeman

Questionmark are partners of SAP, and as well as blogging on the Questionmark site, I also blog on the SAP Community Site.

Sue Martin

I’ve recently started a series of Q&A interviews on thought leaders in the field of assessment and learning within SAP, and last week I interviewed Sue Martin, who is Global Certification Manager at SAP. I thought Questionmark blog readers might be interested in one question I asked her about how SAP ensure their questions are good quality.

John : What effort goes in at SAP to make the questions good quality and set the scoring fairly?

Sue : We follow industry standard practices to get legally defensible exams into place.  In order to get more experiential subject matter expertise, we moved from instructors writing questions to much more involvement from our own consulting organization. At the same time, to get the level of consistency and quality of review, we significantly increased the number of subject matter experts (SMEs) involved.

Here is an overview of the steps we follow:

  1. The first step is job task analysis to identify the tasks that we ask questions on. These are validated with customer organizations, partners and with our own consulting organization.
  2. The next step is the design phase, where we have the item writing workshops that write and review the questions. It depends on the subject matter, but there are normally at least 6 or 7 SMEs that sit in these workshops, supported by a psychometrician.
  3. We then have peer review from another set of SMEs after the workshop. By SMEs I mean primarily people who are actually working in that area, so for a Professional certification that would be a Senior consultant or a Platinum consultant.
  4. Then another group of SMEs go through the exam as beta testers and we look at the results.
  5. Then a group of experts goes through the questions and do a pre-standard setting process, estimating the difficulty level of each question.
  6. Then we have the final standard setting meeting to get to the cut score (pass score) determination phase.  And then at the end of all that, we finally have the exam!

To see the rest of the interview, including questions on how SAP are finding Questionmark effective for authoring and delivery of exams and how they are transforming their certification programme, click here.

Questionmark: a new SAP Partner


john_smallPosted by John Kleeman

I’m pleased to let you know that Questionmark is now an SAP software solution partner.sap_partner_R_tm_p

As you’ll perhaps have seen from SAP’s advertising, many of the world’s best-run businesses run SAP software. Best-run organizations need their employees to perform, learn and be certified, and it makes a lot of sense for us to integrate Questionmark software with SAP so that people can seamlessly move from SAP user interfaces to Questionmark ones and back again.

I’ve had the privilege of working closely with SAP over the last two years. We’ve developed our Connector for use with SAP Learning Solution and deployed it with customers. And I’m now working within Questionmark to encourage deeper links with SAP software in the future so that organizations using SAP will be able to integrate Questionmark assessments even more easily in future.

Questionmark partners with many other great companies, but I’m very proud that we are a formal partner with SAP.

SAP and the SAP partner logo are trademarks or registered trademarks of SAP AG in Germany and in several other
countries all over the world.

Early-birds: Register for the Questionmark 2010 Users Conference by Friday!


Posted by Joan Phaup

We’re fast approaching the first early-bird deadline for the Questionmark 2010 Users Conference. RLB_3816

If you register by this Friday, December 4th, you will save $200 off the full registration fee.  The conference program is taking shape, so check out some  session titles on the schedule-at-a-glance.

Case study topics include Integrating Perception and SAP, Using Flash and Captivate with Perception, and  Distributing the Workload and Increasing Accessibility with Questionmark Live. You will also see some Best Practice and Tech Traning topics listed. There are more to come, so keep checking the site.

The  Conference offers great opportunities for technical training on Questionmark Perception as  well as instruction on how to write better assessments. Delegates will see the latest Questionmark product developments and will influence what happens next. Meeting other Perception Users and learning from their experiences is another major plus of attending the  Conference. There’s really no better place to RLB_4087immerse yourself in learning how to get the most from your assessments.

If you missed our podcast with keynote speaker Dr. David Metcalf, click here and spend a few minutes listening to our conversation about the future of assessment.

….and remember to register! We look forward to seeing you in Miami March 14 – 17, 2010!