The Power of Open: Questionmark’s open assessment platform

Posted by Steve Lay

In the beginning there was CVS, then there was SVN and now there’s Git.  What am I talking about?  These are all source code control systems, systems that are used to store computer source code in a way that preserves the complete version history and provides a full audit trail covering the who, what, when and why changes were made.

When we think of open source software we tend to think of the end product: a freely downloadable program that you can run on your computer or even a complete computer operating system in the case of Linux.  But to open source developers, open source is about more than this ‘free beer’ model of sharing software.  Open source software is shared at the source code level allowing people to examine the way it works, suggest changes to fix bugs, enhance it or even to modify it for their own purposes.  Getting the most from sharing source code requires more than just sharing an executable or a zip file of the finished product, open source developers need to open up their source code control systems too.

For years there have been services that provide a cloud-based alternative to  hosting your own source code.  The SourceForge system enjoyed many years of dominance but more recently it’s advertising sponsored model has seen it fall out of favour.

Most new projects are now created on a service called GitHub, which promises  free hosting of open source projects on a service funded by paying customers who are developing projects privately on the same platform.  The success of GitHub has been phenomenal – Google closed down its own rival service (Google Code) largely because of GitHub’s success.  In fact, GitHub is rapidly becoming a ‘unicorn’ with all the associated growing pains.  GitHub makes it easy to collaborate on projects too with its issue tracking system and user friendly tools for proposing changes (known as ‘pull requests’).

With GitHub as the de facto place to publish and share source code, it makes sense for Questionmark to use it to complement our Open Assessment Platform.  We have published source code illustrating how to use our APIs for many years and even publish the complete source to some of our connectors.  Putting new projects on GitHub means providing sample code in the most transparent and developer-friendly way possible.

Questionmark’s GitHub page lists all the projects we own.  For example, when we first brought out our OData APIs we published the sample reportlet code in the OData Reportlet Samples project.  You can experiment with these same examples running live in our website’s developer pages.

Recently we’ve gone a step further in opening up our assessment platform.  We’ve started publishing our API documentation via GitHub too!  Using a new feature of the GitHub platform we’re able to publish the documentation directly from the source control system itself.  That means you always get access to the latest documentation.

Opening up our API documentation in this way makes it easier for developers to engage with our platform.  Why not check out the documentation project.  If you’re already a GitHub user you could ‘watch’ it to get notified when we make changes.  You can even submit issues or send us ‘pull requests’ if you have suggestions for improvement.

With GitHub as the de facto place to publish and share source code, it makes sense for Questionmark to use it to complement our Open Assessment Platform.  We have published source code illustrating how to use our APIs for many years and even publish the complete source to some of our connectors.  Publishing this source code helps our customers and partners by providing working examples of how to integrate with our platform as well as providing complete transparency for our connectors allowing customers to audit the code before they run it on their own systems.  Putting new projects on GitHub means providing sample code in the most transparent and developer-friendly way possible.

Interact with your data: Looking forward to Napa

Steve Lay HeadshotPosted by Steve Lay

It’s almost time for the Questionmark Users Conference, which this year is being held in Napa, California. As usual there’s plenty on the program for delegates interested in integration matters!

At last year’s conference we talked a lot about OData for Analytics, (which I have also written about here: What is OData, and why is it important? ). OData is a data standard originally created by Microsoft but now firmly embedded in the open standards community through a technical group at OASIS. OASIS have taken on further development, resulting in the publication of the most recent version, OData 4.

This year we’ve built on our earlier work with the Results OData API to extend our adoption of OData to our delivery database, but there’s a difference. Whereas the Results OData API provides access to data, the data exposed from our delivery system supports read and write actions, allowing third-party integrations to interact with your data during the delivery process.

Why would you want to do that?

Some assessment delivery processes involve actions that take place outside the Questionmark system. The most obvious example is essay grading. Although the rubrics (the rules for scoring) are encoded in the Questionmark database, it takes a human being outside the system to follow those rules and to assign marks to the participant. We already have a simple scoring tool built directly in to Enterprise Manager but for more complex scoring scenarios you’ll want to integrate with external marking tools.

The new Delivery OData API provides access to the data you need, allowing you to read a participant’s answers and write back the scores using a simple Unscored -> Saved -> Scored workflow. When the result is placed in the final status, the participant’s result is updated and will appear with the updated scores in future reports.

I’ll be teaming up with Austin Fossey, our product owner for reporting, and Howard Eisenberg, our head of Solution Services, to talk at the conference about Extending Your Platform, during which we’ll be covering these topics. I’m also delighted that colleagues from Rio Salado College will also be talking about their own scoring tool that is built right on top of the Delivery OData API.

I look forward to meeting you in Napa but if you can’t make it this year, don’t worry, some of the sessions will be live-streamed. Click here to register so that we can send you your login info and directions. And you can always follow along with social media by following and tweeting with @Questionmark.

Need Customized Reports? Try Out Our OData API

Sample Attempt Distribution

Sample Attempt Distribution Reportlet

Joan Phaup 2013 (3)Posted by Joan Phaup

The standard reports and analytics that our customers use to evaluate assessment results meet a great many needs, but some occasions call for customized reports. The Questionmark OData API makes it possible to access data securely and create dynamic assessment reports using third-party business intelligence tools.

Once these reports are set up, they provide a flow of data, updating the results as new data becomes available. OData also makes it possible to cross-reference your assessment data with another data source  to get a fuller picture of what’s happening in your organization.

My recent interview with Austin Fossey goes into more detail about this, but you explore this idea yourself thanks to the OData tutorials and dashboards  on Questionmark’s Open Assessment Platform for developers.

The site provides example open source code to show how your organization could provide reportlets displaying key performance indicators from many types of assessment data.  The examples demonstrate these sample reportlets:

  • Attempt Distribution
  • PreTest PostTest
  • Score Correlation
  • Distribution

Questionmark OnDemand customers can plug in their own data to create their own reportlets, and developers can use tutorials to get detailed instructions about connecting to OData, retrieving data and creating charts.

You can also learn a lot about the power of OData at the upcoming Questionmark Users Conference in San Antonio March 4 – 7, so we hope you’ll join us there!

Using OData for dynamic, customized reporting: Austin Fossey Q&A

Joan Phaup 2013 (3)Posted by Joan Phaup

We’ll be exploring the power of the Open Data Protocol (OData) and its significance for assessment and measurement professionals during the Questionmark 2014 Users Conference in San Antonio March 4 – 7.

Austin Fossey, our reporting and analytics manager, will explain the ins and outs of using the Questionmark OData API, which makes it possible to access assessment results freely and use third-party tools to create dynamic, customized reports. Participants in a breakout session about the OData API, led by Austin along with Steve Lay, will have the opportunity to try it out for themselves.

Austin Fossey-42

Austin Fossey

I got some details about all this from Austin the other day:

What’s the value of learning about the OData API?

The OData API gives you access to raw data. It’s an option for accessing data from your assessment results warehouse without having to know how to program, query databases or even host the database yourself. By having access to those data, you are not limited to the reports Questionmark provides: You can do data merges and create your own custom reports.

OData is really good for targeting specific pieces of info people want. The biggest plus is that it doesn’t just provide data access. It provides a flow of data. If you know the data you need and you want to set up a report, a spreadsheet, or just have it in the web browser, you can get those results updated as new data become available. This flow of data is what makes OData reports truly dynamic, and this is what distinguishes OData reports from reports that are built from manually generated data exports.

What third-party tools can people use with the OData API?

Lots! Key applications include Microsoft Excel PowerPivot, Tableau, the Sesame Data Browser, SAP Business Objects and Logi Analytics, but there are plenty to choose from. People can also do their own programming if they prefer.  The website includes a helpful listing of the OData ecosystem, which includes applications that generate and consume OData feeds.

Can you share some examples of custom reports that people can create with OData?

We have some examples of OData reportlets on our Open Assessment Platform website for developers, which also includes some tutorials. I’ve blogged about using the OData API to create a response matrix and to create a frequency table of item keys in Microsoft PowerPivot for Excel. There are so many different ways to use this!

What about merging data from assessments with data from other sources? What are some scenarios for doing that?

It could be any research where you want to cross-reference your assessment data with another data source. If you have another data set and were able to identify participants – say an HR database showing the coursework people have done – you could compare that with their test results to their course activity Reports don’t necessarily have to be about test scores. They can be about items and answer choices – anything you want.

Tell me about the hands-on element of this breakout session.

We will be working through a fairly simple example using Microsoft PowerPivot for Excel  in order to cement the concepts of using OData. We’re encouraging people to bring their laptops with Excel and the PowerPivot add-in already installed. If they don’t have that, they can either work with someone else or watch the exercise onscreen. We will provide a handout explaining everything so they can try this when they are back at work.

What do you want people to take away from this breakout?

We want to make sure people know how to construct an OData URL, that they understand the possibilities of using OData but also the limitations. It won’t be a panacea for everything. We want to be sure they know they have another tool in their tool box to answer the research questions or business questions they encounter day to day.

Our conference keynote speaker, Learning Strategist Bryan Chapman, will share insights about OData and examples of how organizations are using it during his presentation on Transforming Data into Meaning and Action.

Click here to see the complete conference program. And don’t forget to sign up by January 30th if you want to save $100 on your registration.

Transforming Open Data into Meaning and Action: Q&A with Bryan Chapman

Joan Phaup 2013 (3)Learning strategist Bryan Chapman’s keynote address on Transforming Open Data into Meaning and Action will be a highlight of the Questionmark 2014 Users Conference in San Antonio March 4 – 7.

With customers now using the Questionmark OData API to harvest meaning from their assessment results with greater freedom and flexibility, we are excited to hear about the broader implications of open data and the opportunities for learning organizations to make the most of it.

The Open Data Protocol, an industry standard for accessing data sources via the web, provides new opportunities reporting and analyzing assessment results. OData feeds can be consumed and analyzed by many leading business intelligence applications, provide new options for custom reports and dashboards..

I asked Bryan recently about what the advent of OData will mean, particularly with regard to learning and measurement:

The concept of open data is being talked about a lot these days. Why is it important?

Bryan Chapman (2)

Bryan Chapman

I’ve got to confess that I’m a total data junkie, so I get very excited about the endless possibilities of open data. Think of how much data is being collected on a daily basis all over the world from scientific discoveries, government data, opinion polls, and even learning.  Gartner recently said that companies are collecting 300% more data than they did 4 years ago.  It’s crazy. But consider what we can discover by selectively combining data in meaningful ways – something OData enables us to do.

Here’s a non-learning example of how powerful data can be: The CDC (Centers for Disease Control) collects information showing incidence of heart disease on a county-by-county basis. A group called Third World Congress on Positive Psychology developed a way to parse through Twitter feeds across two counties, analyzing the use of 40,000 words in over 80 million tweets.

By combining these separate data feeds, they discovered a pattern between having a positive attitude and having lower risk of heart attack. This is a rather simple example, but just think of what kinds of patterns we will find as open data takes hold.

What is the significance of OData for testing and assessment professionals?

If you want to discover patterns of behavior that make companies successful, what better variable to plug in than testing and assessment data?

Case in point: A while ago I worked with a large software company.  Independent, outside data suggested that customers felt that the company’s help desk support team often lacked knowledge in specific technical areas.  We went in, created an assessment (using Questionmark, by the way!) and created a gap analysis across 60 technical skills areas for all help desk support.

We turned the data into a gap analysis heat map with red, yellow and green indicators showing a range of levels from expertise down to lack of knowledge.  When we presented their senior management with the map, it was very clear to see several things…where training was needed…sometimes having the wrong person on the wrong team, and a whole lot more.

This was great as far as it went, but it was just a single snapshot in time…it wasn’t ongoing. I think back on that project and wonder how much more impressive it would have been if the data was continually measured, linked to a dashboard and frequently compared to the independent audit of customer responses.

OData makes this possible.

How can the use of open data impact learning and performance?

First, with open data, it’s relatively easy to flow the results of learning, testing and assessment right into the performance review process.  I’ve been watching the major talent management vendors who have tools to conduct annual performance reviews, do staff planning, succession planning and pay for performance; many of them are gradually adding OData feeds (both in and out of their systems). So creating that level of interoperability is already starting to shape up. The bigger win is the ability to link learning with organizational performance: Kirkpatrick Level 4!

Most of us feel very comfortable collecting Level 1 and Level 2. Do they like the training?  And are they learning, comparing pre and post test score? Some get how to do level 3 by sending out delayed questionnaires asking what skills are being used on the job; or through performance observation. But open data is really the enabler for linking learning and performance with key company metrics, income, productivity, retention, and lots of bottom line results — especially as other parts of the business make their data available through open channels.

How will your keynote address relate to the specific interests of Questionmark customers?

I’m not a technical guru when it comes to open data…not by a long shot. There will be others there who can tell you all about Questionmark’s OData capabilities. But I really think the hardest part of this is re-imagining how data can be creatively combined to paint the whole picture, or at least understanding what others might do with data that we make available through testing and assessment.

I’ll be sharing several examples of innovative approaches, but that’s just the tip of the iceberg.  If our organizations are really collecting 300% more data than 4 years ago, there are simply way too many data streams to combine.  So we need to start off by keeping things simple – to figure out which data streams can get us where we need to go.

If do my job well, we’ll all start dreaming of ways we can marry data together and apply meaning. And before long, we can expect to see some very creative dashboards linking learning data with actual business performance.

Learn more about the conference program, which includes two new pre-conference workshops: Test Development Fundamentals and The Art and Craft of Item Writing. Register for the conference by January 30 to save $100. Another current learning opportunity: 3-day Questionmark training in Las Vegas February 4 – 6.

New Questionmark OnDemand release enhances analytics and mobile delivery

Jim Farrell HeadshotPosted by Jim Farrell

With Questionmark having just released a major upgrade of our OnDemand platform. I want to highlight some of the great new features and functionality now available to our customers.

Let’s start with my favorite. Questionmark released a new API known as OData, which allows Questionmark customers to access data in their results warehouse database and create reports using third-party tools like PowerPivot for Excel and Tableau. Through a client, a user makes a request to the data service, and the data service processes that request and returns an appropriate response.

You can use just about any client to access the OData API as long as it can make HTTP requests and parse XML responses. Wow…that’s technical! But  the power of the new OData API is that it liberates your data from the results warehouse and lets you build custom reports, create dashboards, or feed results data into other business intelligence tools.


The OData API is not the only update we have made to Analytics. The addition of the Assessment Content Report allows users to review participant comments for all questions within an assessment, topic, or specific question. Enhancements to the Item Analysis report include the ability to ignore question and assessment revisions. This report now also supports our dichotomously-scored Multiple Response, Matching, and Raking question types.

Another improvement I want to highlight is the way Questionmark now works with mobile assessments. An updated template design for assessments when taken from a mobile device embraces responsive design, enhancing our ability to author once and deploy anywhere. The new mobile offering supports Drag and Drop and Hotspot question types — and Flash questions can now run on all Flash-enabled mobile devices.

Click here for more details about this new release of Questionmark OnDemand.