Putting coaching reports front and center

Posted by Joan Phaup

As we approach our first early-bird registration deadline for the Questionmark 2013 Users Conference, I’m checking in with people who will be presenting various sessions during our three days together in Baltimore March 3 – 6.

Abdulquader Kinariwala

It was great to talk the other day with Accenture Certification Manager Abdulquader Kinariwala, who will deliver a case study presentation with his colleague John Kessler, about the use of coaching reports not only to help people succeed in certification exams but also to improve their overall work performance.

Could you briefly describe the role of Questionmark assessments in Accenture’s certification program?

Assessment are very important part of our evaluation mechanism.  When people claim that they are certified – – whether it’s a professional certification or a technology certification  — they must have a credential to prove their qualifications. And to be credentialed, he or she must participate in an assessment.  Assessments are an integral part of our certification process.

I understand you will be zeroing in on the use of coaching reports in your conference presentation. Why are coaching reports important to you?

When we are talking about certifications or learning in general, our assessment framework emphasizes the value of how we coach people to be successful. If we have an employee who is not successful in a certain assessment in a certification program, we share with them the coaching report of their previous performance in an assessments, and we have a discussion about it. With the employee’s permission, we will share the report with his or her career counselor, who will work with the person before they take the assessment again.

Coaching reports are often regarded as a peripheral mechanism for improving performance in an assessment, but we see coaching as a core element of learning.   We use coaching reports not only for failed outcomes but also for partial successes. Someone may have passed an assessment, but there might be a section or two where they could have done better. Our coaching reports provide details about individual topics, so we can pinpoint specific areas where there is room for improvement.

What makes Accenture’s use of coaching reports unique?

We have a customized version of the Questionmark Coaching Report – which we worked on with Questionmark consulting — which allows us to download an Excel extract of the assessment repository. We put that file into one of our custom communication tools which we have created in-house. This tool creates an email that highlights to the person their scores in particular sections and calls out areas where they can improve.

What do you hope people will gain from attending your case study presentation?

We’d like them to get an understanding of how we are using coaching reports,– even for successful candidates — and gain some insights on the custom solution we worked on with Questionmark. We will also have a discussion about how other people are currently using coaching reports, so everyone should have an opportunity to learn from one another.

You’ve attended several Questionmark Users Conference. What do you find most valuable about them?

The ability to network and interact with the Questionmark teams: especially product development, consulting and customer services. I also get a lot out of the keynotes as well as a lot of customer presentations. I like to hear people who have been using Questionmark and consider how to adapt their ideas to our own situation.


Note: The  early-bird Questionmark Users Conference registration discount of $200 ends today!  Check out other conference sessions and click here to sign up.

eAssessment in the Cloud, Sunshine or Thunderstorm?

Sunshine or Thunderstorm?Posted by John Kleeman

Earlier this week, I presented at the online part of the eAssessment Scotland conference on the advantages and disadvantages for academic institutions of using eAssessment in the Cloud “on-demand” or installing it “on-premise” within the institution. Does an on-demand eAssessment service give continual sunshine to a university or college? Or is it safer to install it locally and go on-premise? What questions do you need to ask about the potential thunderstorms using the Cloud?

Questionmark offers both Questionmark Perception, an installable assessment management system, and Questionmark OnDemand, our scalable software-as-a-service system, so we can see the pros and cons of both approaches- and can offer some unbiased advice.

Here is the presentation I gave – you can see it embedded at the end of the post or else view it on the Questionmark Slideshare site.

The presentation suggests that for a university or college, on-demand may be stronger in these areas:

  • Access to innovation
  • Speed/flexibility of deployment
  • Reliability and uptime
  • Scalability
  • Security and cheating
  • Getting IT bandwidth

And that 0n-premise may be stronger in these:

  • Ease of customization/integration
  • Connectivity
  • Governments accessing your data

In these areas, you need to look into the details to determine what would work best in your situation:

  • Data protection
  • Can you change providers?
  • Costs, features and other factors

I believe that for a lot of universities and colleges on-demand offers a lot of value. This is especially so if their IT department is focused elsewhere and does not easily have the bandwidth to manage eAssessment. But it is very important to get your solution right, and if you’re looking at On-demand, you might like to read a paper I presented at the 2012 International Computer Assisted Assessment Conference on How to Decide between On-demand and On-premise eAssessment. This includes a lot of useful questions to ask potential providers when evaluating potential on-demand solutions. You can see the paper here.

I hope you find the presentation useful.

Timing is Everything: Using psychology research to make your assessments more effective

Posted by Julie Delazyn

Over the past year or so, John Kleeman has been sharing new evidence from cognitive psychology indicating that when it comes to making your assessments more effective, timing is crucial.

One of the strongest research findings is that learning is far more effective if it is spaced out over time instead of being delivered all at once. There is evidence that giving regular quizzes helps people by:

  • encouraging them to space out their study in preparation for the quizzes
  • help them learn simply by taking the quizzes and seeing the feedback

Psychology research also shows that taking a quiz or test directly helps you retain information: The more you practice retrieving something, the more likely you are to retain it for the long term.

This SlideShare presentation will help you learn about psychology research and how you can apply it to improve your use of assessments.

Top 5 Questionmark Presentations on SlideShare

Posted by Julie Delazyn

Presentations make up a huge part of the way thought leaders at Questionmark pass on all the great information that comes from our various white papers, research and case studies.

We have been sharing many of these presentations with you by featuring and posting them to our new Questionmark SlideShare page.  I read and answer comments all the time from LinkedIn, Facebook, Twitter and, most recently, Google+ about the value of our presentations as well as the ways in which they are being shared and used. These are in high demand, and last week SlideShare featured Questionmark on its homepage for having the most popular presentation on that particular day.

So, we thought it would be helpful to highlight our five most popular presentations on our SlideShare page (in no particular order):

Feel free to comment, share and let us know in which ways these have helped you!

A Postcard from Preston

Posted by Steve Lay

Recently I attended the UK Academic briefing at the University of Central Lancashire, in Preston, UK.  It kicked off with Joseph Kelly, one of our Account Managers, giving an overview of some key technologies that make up the Questionmark assessment platform.

I also got a chance to update everyone on some of our current projects.

To finish off the event, John Dermo talked about the University of Bradford’s approach to formative assessment using Questionmark Perception.  I always find John’s talks interesting and, as usual, he provided plenty of pictures to help us visualise the data captured during his research.

John Dermo

His analysis looked in depth at student use of formative feedback, one of the conclusions being that “students who view the formative [e-assessments] and feedback as part of their learning show the greatest amount of progress”.

The slides from John’s presentation at the briefing are now available here.

I’d like to thank our hosts for providing an excellent venue and everyone who attended for making this such an interesting event.

Observational assessments: some learning resources

julie-smallPosted by Julie Delazyn

Observational assessments, in which an observer assesses a participant and rates his or her performance, make it possible to evaluate skills or abilities that  are difficult to measure using “traditional” assessments. The ability to deliver observational assessments using mobile devices is opening up new possibilities in many different settings:

  • Certifications: Observational assessment is often a key component of the certification process – in some cases required by regulatory authorities.  For an example of this, see our recent  case study, Covidien Delivers Product Knowledge Tests and Observational Assessments on the Apple iPad.
  • Medical and dental schools – for  examinations such as the Objective Structured Clinical Examination (OSCE).
  • Equipment operation – for recording and rating operators’ skills, abilities, safety practices and adherence to required procedures.
  • Customer service: Workplace assessments are used to evaluate the performance of retail sales and customer service staff and to determine  where additional training is required.
  • Driving tests: Examiners observe and rate license applicants on  parking, observation, awareness and adherence to rules of the road.
  • Performance or Level 3 assessment: Observing and monitoring behavioral changes in the workplace before and after training helps learning organizations  evaluate how well participants apply what they have learned during training after they are back on the job.

If you would like to learn more about Observational Assessments,  check out this Questionmark presentation on SlideShare.