Early-bird savings on conference registration end today: Sign up now!

Joan Phaup 2013 (3)Posted by Joan Phaup

Just a reminder that you can save $200 if you register today for the Questionmark 2014 Users Conference.

We look forward to seeing you March 4 – 7 in San Antonio, Texas, for three intensive days of learning and networking.

Check out the conference program as it continues to take shape, and sign up today!

This conference truly is the best place to learn about our technologies, improve your assessments and discuss best practices with Questionmark staff, industry experts and your colleagues. But don’t take my word for it. Let these attendees at the 2013 tell you what they think:



Questionmark Live: Watching the numbers grow

Jim Farrell HeadshotPosted by Jim Farrell

I hope everyone is having a wonderful holiday season.

Typically I come to you telling you about amazing new features, but this time I want to talk about the number of items created in Questionmark Live over the past year.

Let me set the stage. We started the year with approximately 30,000 questions being created in Questionmark Live each month.
As part of the team that gave birth to our newest authoring tool, I was over the moon about his strong start. This proved to me that the features we were releasing were helping people build up their item banks. I was excited enough as it was, but then came September, October and November.

September saw or most impressive increase in usage. More than 97,000 questions were created — almost 100,000 questions in just one month.

October followed with more than 74,000 questions –- including more than   20,000 questions on October 16th alone. This is the number our development teams were most excited about: 20,000 questions!  And the system ran flawlessly, speaking to the scalability of the software and the OnDemand Platform.

November proved that the usage was legitimate. More than 121,000 questions were created that month.

It’s clear now that Questionmark Live is the preferred authoring tool among Questionmark users — with ease of use and scalability as its foundation.

You didn’t think I would end without talking about something new, did you?

I can’t resist telling you that you no longer need to approve authors to have access to Questionmark Live. Anyone with a valid Questionmark Communities account can gain access to Questionmark Live. We hope this makes it easier for you to crowd- source content within your organization and write good questions to solve real business problems.


The Questionmark 2014 Users Conference will include bring-your-own-laptop sessions on Creating Items and Topics as well as Collaborative Assessment Authoring in Questionmark Live. The early-bird registration discount of $200 is available through tomorrow December 12th, so sign up now!

Information Security: from Human Factors to FIPS

Steve Lay HeadshotPosted by Steve Lay

Last year I wrote about Keeping up with Software Security and touched on the issue of cryptography and the importance of keeping up with modern standards to ensure data is stored and communicated securely. In this post, I return to this subject and dive a little deeper into maintaining information security and the role that software plays in it.

The first thing to is that software can only ever be part of an information security strategy. It doesn’t matter how complex your password policy or how strong your data encryption standards are if you allow authorised individuals to share information inappropriately. Amongst all the outcry following the information disclosed by Edward Snowden, it is surprising to me that the focus hasn’t been on why an organization that plays such a crucial role in security (in its widest sense) was so easily undone by a third-party contractor. According to this article from Reuters news agency, one of the techniques Snowden used to gain access to information was simply asking co-workers to give him their passwords.

The computing press will sometimes try and persuade you that there is a purely technical solution to this type of problem. It reminds me of the time my son’s school introduced fingerprint recognition for canteen payments, in part, to allow parents to check up on what their children had been spending — and eating!  It wasn’t long before the phrase, “Can I borrow your finger?” was commonplace amongst the pupils.

I hope my preamble has brought home to you the basic idea that these ‘human factors’, as the engineers like to call them, are just as important as getting the technology right. Earlier this week I had to go through my own routine security testing as part of our security process here at Questionmark, so this stuff is fresh in my mind!

Security Standards

Information security, in this wider context is covered by a whole series of international standards commonly known as the ISO 27000 series.  This series of standards and codes of practice cover a wide range of security processes and computer systems. To an engineer looking for a simple answer it can be frustrating though. ISO 27002 contains advice on the use of cryptography, but it runs more like a policy checklist. It won’t tell you which algorithms are safe to use. In part, this is a recognition of how dynamic this field is. The recommendations might change too fast for something like an ISO standard which takes a long time to develop and is designed to have a fairly long shelf-life.

For more practical advice to engineers developing software, help is at hand from the Federal Information Processing Standards (known as FIPS).  FIPS was developed by the U.S. government to help fill out the gaps where externally defined standards weren’t available. It ranges across many areas of information processing, not just security, but one of the gaps FIPS fills is specifying the details of which cryptographic algorithms are fit for modern software and, by implication, which ones need to be retired (FIPS-140). This standard has become so important that the word FIPS is often used to refer only to FIPS-140! It isn’t restricted to the U.S. either; it is being freely adopted by other governments including my own government here in the UK.

FIPS 140 also has a certification programme. The purpose of the programme is to certify the implementation of cryptographic code to check that it does indeed correctly implement the security standard. Microsoft have a technical article to explain which parts of their platform and which versions have been certified. There is even a “FIPS mode” in which the system can be instructed to use only cryptographic algorithms that have been certified.

Concentrating the cryptographic features of an application into a small number of modules that are distributed with the underlying operating system rather than having each application developer individually implement or incorporate the code themselves will, over time, make it easier to make use of appropriate cryptography and to implement policies such as those described by ISO 27002.

Automating the creation of customized assessment completion certificates – Q&A

Joan Phaup 2013 (3)Posted by Joan Phaup

I spoke recently with Justin Devine, CIO of The Aurelius Group, a consulting company that works with federal government agencies. Justin and the company’s system administrator, Brandon Funkhouser, will be among the case study presenters at the Questionmark 2014 Users Conference in San Antonio, Texas, March 4 – 7.

The Aurelius Group’s client, a large federal agency, required a solution that would allow the delivery of web-based assessments to a large workforce in support of these objectives:

•    Provide individuals with an objective assessment of their individual strengths and weaknesses coupled with a recommended list of targeted learning interventions based on assessment results and tailored to the individual’s development needs.
•    Provide agency leadership with aggregate reports revealing the strengths and weaknesses of the workforce as a whole.

I’m looking forward to the case study presentation about this and asked Justin for some background on it.

Justin Devine1

Justin Devine

What’s the Aurelius Group?

We are a Service-Disabled Veteran-Owned Small Business (SDVOSB) that provides services primarily to the federal government.  We offer management consulting with a focus on acquisition and procurement solutions, human capital analytics, and IT solutions around the Microsoft Office 365 platform.

How do you use Questionmark assessments?

One of our principal projects uses Questionmark Perception to deliver assessments to a federal workforce of approximately 4500.  Charged with managing the Federal Government’s buying activities, these employees require a great deal of knowledge about approved regulations, procedures, and rules governing the federal acquisition and procurement process. The cornerstone of this knowledge, spanning over 2000 pages, is the Federal Acquisition Regulation.

Considering the breadth of knowledge required of federal acquisition workers, the capability to regularly measure and respond to individual strengths and weaknesses allows participants to focus their development efforts on the areas that are most valuable for each individual. To facilitate this, our program uses diagnostic assessments to provide Targeted Training Recommendations (TTRs) to participants based on topic level benchmarks.  After completing an assessment we use the outcomes to direct them to courses that will shore up their weaknesses. For example, if a user doesn’t reach the cut score for a particular competency, they are immediately led to targeted training recommendations that would help them improve in that area.  We also send them their topic results table, which contains the training recommendations, by email.

How else do you use the information from diagnostic assessments?

We also roll up all the data and give our client information about the entire workforce and subsections of the workforce.  Along with its inclusion in the agency’s strategic human capital plan, leaders can use the data to adjust their HR strategy for hiring, selection and promotion. They can adjust their strategy to leverage their employees’ strengths and correct or minimize the impact of their weaknesses.  If you have a bunch of workers who are strong some areas, you can use that information to balance your hiring criteria. You can also take advantage of the strengths people have instead of just reacting to their weaknesses. This can bring about more efficient use of resources and a better performing workforce.

So where do the completion certificates come in?

To encourage participation in the assessment and the recommended learning opportunities, the agency offers continuing education credits to participants.  In addition, participants are getting recall practice while they take these tests and reinforcing their learning so they deserve some learning credit.  These tests last four to five hours, so people need some encouragement and incentive to take them!  To provide participants with proof of participation, we developed a solution for generating custom completion certificates.

What are you looking forward to at the conference?

This is the first time Brandon and I are participating as presenters, so that’s really exciting for us. Last year we were consumers. Now we hope we will be in a position to help others.

Congratulations to Anne Dunsmore of CFA Institute on winning a free conference registration in our sweepstakes on Facebook!

You have the opportunity to save $200 on your own conference registration: Just sign up by December 12th to receive this special early-bird discount.

Integrating and Connectors – Cornerstone OnDemand

Doug Peterson HeadshotPosted By Doug Peterson

In my last Integrating and Connectors blog post, I noted that there are two ways that Questionmark can integrate with other systems:
1.    Using a dedicated connector, or
2.    By publishing an assessment as a SCORM or AICC content package and importing that package into your LMS.

Over the next several blog posts, I will present short videos giving examples of different integration approaches. Let’s start by looking at publishing a SCORM or AICC content package from Questionmark and importing it into an LMS, in this case, Cornerstone OnDemand.

This approach will work with just about any quality LMS available. The particulars of importing and setting up a content package will differ from LMS to LMS, but the overall approach and benefits are the same.


Psychometrics and Measurement Design: A conversation

Joan Phaup 2013 (3)Posted by Joan Phaup

Many delegates to the Questionmark 2014 Users Conference in San Antonio March 4 – 7 want to learn about assessment-related best practices.

Austin Fossey, our Reporting and Analytics Manager, will talk about Principles of Psychometrics and Measurement Design during one of the many breakout sessions on the agenda.

Austin Fossey

Austin Fossey

Austin had just joined Questionmark when he attended the 2013 conference. This time around, he’ll be more actively involved in the program, so I wanted to learn more about him and his presentation plans.

What made you decide to study psychometrics?

I was working in customer service at a certification testing company. They always brought in psychometricians to build their assessments. I’d never heard of psychometrics before, but I had studied applied math as an undergraduate and thought the math behind psychometrics was interesting. I liked the idea of doing analytical work and heard that psychometricians are always in demand, so I got started right away studying educational measurement at the University of Maryland.

How do you make principles of psychometrics understandable to, well, mere mortals?

I don’t think psychometricians are different than anybody else. Most of it is applying a probabilistic model to a set of data to make an inference about an unobserved trait. Those models are based on concepts or theories, so you don’t have to explain the math as long as you can explain the theory. People understand that.

I really like evidence-centered design, because it provides principles and a vocabulary that can be used by everyone involved in assessments. Using this framework, psychometricians can communicate about measurement design with subject matter experts, item writers, curriculum specialists, programmers, policy makers — all the stakeholders, from start to finish.

Who do you think would benefit from attending your presentation about psychometrics and measurement design?

People who feel they are applying the same test development formula day in and day out and who wonder if there might be a better way to do it. Even with certifications, which usually follow excellent standards based on best practices, we should always be critical about our assessments and we should always be aggressive about ensuring validity. It would be great to see people there who want to be mindful of every decision they make in assessment design.

How could people prepare for this session?

I hope they bring examples of their own test development process and validity studies. We can discuss people’s own experiences and the hurdles they have faced with their measurement design. Other than that I would say just bring an open mind.

What would you like your audience to take away from your presentation?

People who may be new to measurement design and psychometric concepts like validity can take away some tools to use in their assessment programs. I hope that if more experienced people come, they can learn from each others’ experiences and go away with new ideas about their own approach to assessment design.

What do you hope to take away from the Users Conference?
I want to harvest a lot of feedback with our clients during conversations and focus groups, so that we can recalibrate ourselves for the work we are doing and prioritize our tasks.

The session on psychometrics is just one of several for Austin in 2014. Check out the conference program and register by December 12 to save $200.