Test your Valentine’s Day knowledge

julie-smallPosted by Julie Delazyn

In the spirit of Valentine’s Day we wanted to test your knowledge on this holiday with a short trivia quiz.  It took me less time to write the questions using Questionmark Live than it did to do my research. We hope this fun quiz will teach you something new about this holiday. Enjoy your day!

Podcast: Streamlining sales training and regulatory compliance with online assessments


Posted by Joan Phaup

Scott Hertzberg

Pacific Pulmonary Services specializes in oxygen therapy for patients to use in their own homes. The company needs to ensure that sales people  understand all the different aspects of its products. And the company’s delivery technicians must comply with FDA and Medicare guidelines for transporting oxygen — a hazardous material — and working with it safely. Regulations of state boards of pharmacy as well as fire marshals and environmental health departments must be followed, too.

These two needs — for effective sales training and w ell documented regulatory compliance — drive Pacific Pulmonary’s use of online assessments. During my recent conversation with Scott Hertzberg, the company’s licensing coordinator, I learned how centralizing the administration of assessments has brought savings in time and money.

Questionmark’s first Assessment Innovation Award goes to…

Posted by Joan Phaup

Having admired for many years the creativity and can-do spirit of our customers, we’re pleased to announce the first-ever Questionmark Assessment Innovation Award, which we’ll present at the  Questionmark Users Conference in Los Angeles next month.

For this year’s honor, we’ve chosen Accenture for their work with more than 30 internal certification programs that combine training, experience  and (of course!) assessments – mostly medium- to high- stakes online tests as well as work product evaluations and interviews.

Allison Horn

Over the past three years, Accenture has moved ahead on many fronts to grow their internal assessment development and delivery capabilities to meet the increased demand for knowledge and skill assessments while upholding high quality standards. Moving to a three-tier assessment model, the company  has modified their test development processes so that the amount of time, money and effort invested in each test is determined by its overall stakes. Foundation level programs, for instance, now require significantly less  subject matter expert time, beta testing and item analysis than master level programs. In addition to more efficient development processes, the company is using new tools to automate item and test analysis more than ever before.

On the delivery side, Accenture has begun using testing kiosks for smaller locations and adopting standard models for reporting results. They’ve also enhanced the use of their Questionmark/LMS integration to give certification candidates a more consistent, familiar end-user experience.

Allison Horn,  Senior Manager – Global HR  at Accenture, will accept the award on Accenture’s behalf  in addition to co-presenting a case study on Enabling Self-Service Reporting. Two of her colleagues will be talking at the conference about Accenture’s Tiered Assessment Model for Internal Certification.

You can see the entire list of case study presentations, peer discussions, demos, best practice presentations, product training breakouts and general sessions, by clicking here. And you can register online now!

Your chance to comment on Standards for Educational and Psychological Testing

greg_pope-150x1502Posted by Greg Pope

The Standards for Educational and Psychological Testing are considered the gold standard for good practice testing and assessment. The document is produced by 3 bodies working together: the  American Educational Research Association (AERA), the American Psychological Association (APA) and the National Council on Measurement in Education (NCME), and is widely referenced by testing and assessment professionals in the USA and internationally seeking to follow best practice. I have a copy on my bookshelf and check it frequently.

The Standards were last published in 1999 and are now going through a revision process to take account of technology, legal and assessment developments in the last decade. The committee producing the new Standards has made a  draft of the new standards and they are consulting for review and comments. You can register to see the draft standards and make comments at www.teststandards.net. There are Chapters on:

  • Validity
  • Errors of Measurement and Reliability/Precision
  • Fairness in Testing
  • Test Design and Development
  • Scores, Scales, Norms, Cut Scores, and Scores Linking
  • Test Administration, Scoring and Reporting
  • Supporting Documentation for Tests
  • The Rights and Responsibilities of Test Takers
  • Test Users’ Rights and Responsibilities
  • Psychological Testing and Assessment
  • Testing in Employment and Credentialing
  • Educational Testing and Assessment
  • Uses of Testing for Policy Purposes

Documents like this benefit from the widest possible review by stakeholders. The revised Standards are likely to significantly influence the assessment world for the next decade, so if you are interested, it’s a good time to review at www.teststandards.net. The deadline for comments is April 20th, 2011.

Using Survey Data to Dynamically Generate P.A.C.E. Certificates

Posted by Joan Phaup

Organizations that offer continuing education programs know too well the effort and cost of printing and mailing the certificates that many people rely on to provide proof of their successful participation in continuing education programs.

Beckman Coulter, Inc., which develops, manufactures and markets diagnostic systems for biomedical testing, uses Webinars and other training events to keep medical technicians up to date with hematology, chemistry, hemostasis and other subjects pertinent to screening blood samples. Like so many other organizations, the company used to print out P.A.C.E. (Professional Acknowledgment for Continuing Education) certificates and mail them back to each person. But what if it were possible to harvest data from post-training evaluation surveys and and automatically generate the certificates using data gathered from post-seminar assessments?

A new case study on our Web site tells how a custom solution built upon Questionmark’s Open Assessment Platform has made it possible for the company to capture key information about participants and use it present each person with links to the particular certificates they require. In turn, the participants can  print and save their certificates themselves, thereby eliminating the need to print and mail certificates.

Delegates to the Questionmark Users Conference in Los Angeles March 15 -18 will have the opportunity to drill down into the workings of this solution during a  presentation by Beckman Coulter Technical Training Supervisor Frank Loforte. You can click here to see all the conference sessions.

Knowledge-Check Assessments within Software User Assistance and Documentation

Posted by John Kleeman

We’ve been advocating for our customers to embed knowledge checks within learning, and I’m glad to say that we have been doing this ourselves. As we say in the software industry when a company uses its own products, we’re eating our own dog food!

The evidence shows that you learn more if you study and take a quiz than if you just study and study, so we wanted to give this benefit to our users.

Questionmark has an extensive knowledge base of 600+ articles, which supplement our user manuals. These knowledge bases require registration to view, but here is an example knowledge base that is free for all to view.  Our knowledge checks typically ask 3 to 5 questions and are randomized so you get different questions when you come to the page again. We’ve put knowledge checks within the most popular articles, and since these have now been live for several months we can share some of the results:

  • On average, 13% of visitors to our knowledge base pages with embedded knowledge checks answer the questions and press Submit to see the results and feedback.
  • The response rate varies considerably depending on the subject matter from 2% in a few technical articles to over 50% in a few where the knowledge check is very appropriate.
  • About 60% of participants get a score of 75% or higher.

Here is some advice from  our documentation team lead (Noel Thethy) and me on what we’ve learned about knowledge checks in user assistance:

  1. Focus knowledge checks in articles that give learning in areas people want to learn for the long term. We found few people clicked on the knowledge checks in areas like installation advice, where you just want to do something once, but there was more interest in articles that explained concepts.
  2. Embed knowledge checks in prominent locations within content so that people can see them easily.
  3. Align questions with key learning points.
  4. Ensure the vocabulary within a knowledge check is consistent with the information it pertains to.
  5. Provide meaningful feedback to correct misconceptions.
  6. Review the questions carefully before publishing (Questionmark Live is great for this).
  7. Plan for regular reviews to make sure the content remains valid as your software changes.
  8. Use references or a naming convention to ensure it is easy to associate knowledge checks with articles in reporting.
  9. Unless you want to capture individual results, use a generic participant name to make filtering and reporting on results easier.
  10. Use the Assessment Overview report to get an overview of results and the Question Statistics or Item Analysis reports to identify which questions people are weak at; this may show that you need to improve your learning material.

I’d love to hear any questions or comments from anyone interested in knowledge checks in user assistance, feel free to email me at john@questionmark.com. To give you a flavour of how a knowledge check helps you practice retrieval on something you’ve learned, answer the three questions on this page to check your knowledge on some of the above concepts.