What do you want for the holidays?

Posted by Howard Eisenberg

All I want for the holidays is …

As the acting product owner for Questionmark’s Reporting and Analytics zone, I would love to hear how you would complete that sentence … with respect to assessment reporting and analytics, of course.

To help stimulate the ideas, I will highlight recent developments in our reporting and analytics suite.

The Introduction of the Results Warehouse and Questionmark Analytics

In version 5.3, we introduced the Results Warehouse. This is a database of assessment results that is separate from the database responsible for delivering assessment content and storing the participant’s responses. Results are extracted, transformed and loaded (ETL’ed) into the Results Warehouse from the delivery database on a recurring schedule. This database is the data source for the Questionmark Analytics reports.

With the advent of Analytics, we have introduced some new reports and we plan to continue building reports in Analytics. In the case of the Item Analysis report, we’ve actually ported that to Analytics entirely, and in so doing have delivered improvements to the visualization of item quality and report output options.

Addition of New Reports in Questionmark Analytics

Here’s a brief inventory of the reports currently available in Analytics. You can read-up on the purpose of each of these reports and see sample outputs by consulting the Analytics Help Center.

In the spirit of holiday gift-giving, allow me to expound on a few of these reports.

Results Over Time and Average Score by Demographic

These are two separate reports but they are similar in that each one displays an average assessment score within the context of a 95% confidence interval, and a count of the number of results (sample size).

The “Results Over Time” report plots the assessment mean over a period of time, the interval of which is selected by the user.

The “Demographic” report does the same in terms of displaying a mean score, but it groups the results by a demographic. In this way, it enables the report consumer to compare the mean across different demographic groups.

Assessment Completion Time

This report can be used to help investigate misconduct. It plots assessments results on a score axis and a completion time axis. Outliers may represent causes of misconduct. That is, if a participant scores above the mean, yet takes an abnormally short time to complete the assessment; this may represent a case of cheating. If a participant takes an abnormally long time to complete the assessment, yet scores very poorly; this may represent a case of content theft. The report allows the user to set the range for normal score and completion time.

Item Analysis

Finally, the item analysis report has been improved to provide users with better visualization of the quality of items on an assessment form, as well as more output options.

Suspect items are immediately visible because users can specify acceptable ranges for p-value and item-total correlation. Items that fall within acceptable ranges for each measure are green, those that fall outside of the acceptable range for one of the two measures are orange, and any that miss the mark for both p-value and item-total correlation are red.

Additionally, different sections and level of detail included in this report can be output to PDF and/or comma separated value files.

So … what’s on your wish list for the holidays?

4 Responses to “What do you want for the holidays?”

  1. Valerie says:

    Is there a report that will allow me to compare 2 individual results? For example, some amount of time after taking training, both the supervisor and the employee take an assessment about the impact of the training on the employee’s job performance and organizational outcomes.

  2. Howard Eisenberg Howard says:

    Valerie,
    It sounds like you are trying to execute a level 3 type evaluation, by taking a 180-degree view of perceived job performance. Questionmark software can indeed be used in this fashion but there isn’t a specific report that can provide a turnkey comparison.

    This type of reporting is only possible when the relationship between the learner (subject) and the learning event (training), and the learner and the learner’s direct supervisor is known and available to the reporting system.

    More often than not, these type of contextual, organization data do not reside in Questionmark but rather in an HR or other learning system. Thus to do a level 3 / 180-degree type of assessement typically requires a well-controlled evaluation process (for instance subjects may need to self-assess first and only after the self-assessment is complete can the supervisor assess the subject) and often some consulting.

    That said, Questionmark is keen to better understand what that business process looks like, with the aim toward bringing a focused solution to market for level 3’s or 180’s.

    I would welcome the opportunity to discuss this in more detail with you.

    Sincerely,
    -Howard

  3. David Glow says:

    Actually, the reporting tool itself, no… but, if you export to data at certain points, or extract two separate individuals, you can import into many analytics tools (sometimes as simple a Excel cross-tabbing can do the trick).

    I have found that the Export to Excel (or other export options) are very good when wanting to compare assessment or survey results data with information that come from our business applications (sales, quality metrics…). A nice way to measure potential training impact (an employee’s potential skill to be applied to business) to the results experienced in business.

    Really nice options to get data from two different systems that should be speaking to dance a bit.

    Might be an option for you.

  4. Howard Eisenberg Howard says:

    David,
    Thanks for joining the conversation. I detect a bit of honest sarcasm in that first remark, and trust me I know where the sentiment comes from.

    Your example of correlating assessment data with other business metrics is really where Questionmark and assessment in general can add business value. It’s very refreshing to hear an experienced practitioner like you to speak like this.

    Questionmark will be releasing a new API in Q1 of this year – the OData Results Feed – that will make it possible to consume results data via a RESTful API. This will enable ad hoc reporting in “desktop” BI tools such as PowerPivot for Excel or Tableau. It will also enable the constant feeding of a enterprise data warehouse where the mash-up of assessment and other business data can be turned into business intelligence. I am very eager to hear from customers who start using the OData feed for this type of reporting.

Leave a Reply