Internet assessment software pioneer Paul Roberts to retire

Paul Roberts photoPosted by John Kleeman

We think of the Internet as being very young, but one of the pioneers in using the Internet for assessments is about to retire. Paul Roberts, the developer of the world’s first commercial, Internet assessment software is retiring in March. I thought readers might like to hear some of his story.

Paul was employee number three at Questionmark, joining us as software developer in 1989 when the company was still working out of my home in London.

During the 1990s, our main products ran on DOS and Windows. When we started hearing about the new ideas of HTML and the web, we realized that the Internet could make computerized assessment so much easier. Prior to the Internet, testing people at a distance required a specialized network or sending floppy disks in the mail (yes people really did this!). The idea that participants could connect to the questions and return their results over the Internet was compelling. With me as product manager, tester and documenter for our new product — and Paul as lead (and only!) developer — he wrote the first version of our Internet testing product QM Web, which we released in 1995.

QM Web manual cover

QM Web became widely used by universities and corporations who wanted to deliver quizzes and tests over the Internet. Later in the nineties, learning from the lessons of QM Web, we developed Questionmark Perception, our enterprise-level Internet assessment management system still widely used today. Paul architected Questionmark Perception and for many years was our lead developer on its assessment delivery engine.

One of Paul’s key innovations in developing Questionmark Perception was the use of XML to store questions. XML (eXtensible Markup Language) is a way of encoding data that is both human-readable and machine-readable. In 1997, Paul implemented QML (Question Markup Language) as an early application of this concept. QML allowed questions to be described independently of computer platforms. To quote Paul at the time:

“When we were developing our latest application, we really felt that we didn’t want to go down the route of designing yet another proprietary format that would restrict future developments for both us and the rest of the industry. We’re very familiar with the problems of transporting questions from platform to platform because we’ve been doing it for years with DOS, Windows, Macintosh and now the Web. With this in mind, we created a language that can describe questions and answers in tests, independently of the way they are presented. This makes it extremely powerful because QML now enables the same question database to be presented no matter what computer platform is chosen on or whatever the operating system.”

Questionmark Perception and Questionmark OnDemand still use QML as their native format, so that every single question delivered by Questionmark technology has QML as its core. QML was very influential in the design of the version 1 IMS Question & Test Interoperability specification (IMS QTI), which was led by Questionmark CEO Eric Shepherd and to which Paul was a major contributor. Paul also worked on other industry standards efforts including AICC, xAPI and ADL SCORM.

Over the years, many other technology innovators and leaders have joined Questionmark, and we have a thriving product development team. Most members of our team have had the opportunity to learn from Paul over the years, and Paul’s legacy is in safe hands: Questionmark will continue to break new frontiers in computerizing assessments. I am sure you will join me in wishing Paul well in his personal journey post-retirement.

Next Generation Assessment Technology & Exciting Events Driving the Conference Agenda

Now that we have the program in place for Questionmark Conference 2017, I’m eager to highlight a few sessions that you will have a chance to attend in Santa Fe, New Mexico March 21-24.

Before the conference gets rolling, there are two full-day workshops available Tuesday, March 21:

Here’s a peak and the agenda. You can explore the entire list of sessions here: Conference Program.

Questionmark Features & Functions

Case Studies

Best Practices

Networking Events

We have some fantastic networking events planned as well.

We’re kicking off the conference with our signature dessert reception. The next day, you will have a chance to enjoy Santa Fe enchanting downtown and dine with a group of fellow assessment professionals. But it’s all culminating to our final event: Meow Wolf’s House of Eternal Return – an exciting multimedia immersive art exhibit experience. Watch out for surprise acts! We look forward to an evening of eating, networking and celebrating!

Questions? Email conference@questionmark.com. We’re happy to help!

How to manage compliance in a highly regulated world? [30-minute webinar]

Posted by Chloe Mendonca

If your industry demands compliance, then your people need compliance training and certifications. Learning and training records are almost always reviewed during regulatory audits to check that employees have received the required training and their competencies or certifications are up to date and valid. The regular assessment of employee knowledge and competencies ensures you’re always ready for an audit or if something goes wrong.

Did you know…


Source: Brandon Hall Group

Perhaps you already understand the value assessments can bring, but need to convince your management team. Or perhaps you’d like a better view on how to use assessments most effectively to ensure compliance. You’re not alone.

Join us for a 30-minute webinar on Thursday March 9, 2017 to:

  • discuss the critical role assessments play in compliance learning
  • explore the benefits of using assessments before, during and after training
  • find out 7 ways assessments fortify compliance
  • get best practice tips for ensuring valid and reliable assessments

We also have several other webinars you may be interested in:

  • How to write high-quality test items – March 4, 2017

If you’d like best practice tips to improve your test items and ensure they produce fair, valid and reliable results then sign up for this 30-minute webinar.

  • Introduction to Questionmark’s Assessment Management System – Various dates and times

Get an overview of Questionmark’s features and functions in this live demo. We will look at the basics of authoring, delivering and reporting on surveys, quizzes, tests and exams.

Early-Bird Conference Special Ends Today

Questionmark Conference 2017 is where you will want to be for three intensive days of learning and networking in Santa Fe, New Mexico March 21-24.

Early-bird registration ends TODAY, February 22. This is your last chance to save $100 on this outstanding professional development opportunity.

Why attend?

  • Learn best practices for assessment authoring, delivery and reporting
  • Make the most of Questionmark features and functions
  • Meet with Questionmark technicians and get answers to your questions
  • Influence the product roadmap
  • Network with your peers

See the full conference program & register by the end of the day for early-bird savings!

 

Final Early-Bird Discount Ends Next Week!

Register for Questionmark Conference 2017 by Wednesday, February 22, for a chance to take advantage of of our final early-bird discount.

Register now and save!

Here are some highlights from the agenda:
  • Assessments as Career Development Tools at Caterpillar Inc.
  • BYOL: Next Generation Assessment Authoring Using Demographics and Jump Blocks
  • Secure Authentication: Single sign-on for Questionmark
  • Staying Ahead of Evolving Security Threats
  • Setting a Fair, Defensible Cut (Passing) Scores

Don’t miss out on the opportunity to:

  • Maximize your Assessment Value
  • Harness the Power of your Results
  • Innovate with us on Future Assessment Technologies

Register now for early-bird savings
Book your room at the Inn and Spa at Loretto before Thursday, Feb. 16 to take advantage of our special rates

We look forward to seeing you in Santa Fe, New Mexico, March 21-24 for our most important learning event of the year!

Seven tips to recruit and manage SMEs for technology certification exams

imagePosted by John Kleeman

How do you keep a certification exam up to date when the technology it is assessing is changing rapidly?

Certifications in new technologies like software-as-a-service and cloud solutions have some specific challenges. The nature of the technology usually means that questions often require very specialist knowledge to author. And because knowledge of the new technology is in short supply, subject matter experts (SMEs) who are able to author and review new items will be in high demand within the organization for other purposes.

Cloud technological offerings also change rapidly. It used to be that new technology releases came out every year or two, and if you were writing certification exams or other assessments to test knowledge and skill in them, you had plenty of notice and could plan an update cycle. But nowadays most technology organizations adopt an agile approach to development with the motto “release early, release often”. The use of cloud technology makes frequent, evolutionary releases – often monthly or quarterly – normal.

So how can you keep an exam valid and reliable if the content you are assessing is changing rapidly? Here are seven tips that could help – a few inspired by an excellent presentation by Cisco and Microsoft at the recent European Association of Test Publishers conference.

  1. Try to obtain item writing SMEs from product development. They will know what is coming and what is changing and will be in a good position to write accurate questions. 
  2. Also network for SMEs outside the organization – at technology conferences, via partners and resellers, on social media and/or via an online form on your certification website. A good source of SMEs will be existing certified people.
  3. Incentivize SMEs – what will work best for you will depend on your organization, but you can consider free re-certifications, vouchers, discounts off conferences, books and other incentives. Remember also that for many people working in technology, recognition and appreciation are as important as financial incentives. Appreciate and recognize your SMEs. For internal SMEs, send thank you letters to their managers to appreciate their effort.
  4. Focus your exam on underlying key knowledge and skills that are not going to become obsolete quickly. Work with your experts to avoid items that are likely to become obsolete and seek to test on fundamental concepts not version specific features.
  5. When working with item writers, don’t be frightened to develop questions based on beta or planned functionality, but always do a check before questions go live in case the planned functionality hasn’t been released yet.
  6. Analyze, create, deliverSince your item writers will likely be geographically spread and will be busy and tech-literate, use a good collaborative tool for item writing and item banking that allows easy online review and tracking of changes. (See https://www.questionmark.com/content/distributed-authoring-and-item-management for information on Questionmark’s authoring solution.)
  7. In technology as in other areas, confidentiality and exam security are crucial to ensure the integrity of the exam. You should have a a formal agreement with internal and external SMEs  who author or review questions to remind them not to pass the questions to others. Ensure that your HR or legal department are involved in the drafting of these so that they are enforceable.

Certification of new technologies helps adoption and deployment and contributes to all stakeholders success. I hope these tips help you improve your assessment programme.

« Previous PageNext Page »