The Power of Open: Questionmark’s open assessment platform

Posted by Steve Lay

In the beginning there was CVS, then there was SVN and now there’s Git.  What am I talking about?  These are all source code control systems, systems that are used to store computer source code in a way that preserves the complete version history and provides a full audit trail covering the who, what, when and why changes were made.

When we think of open source software we tend to think of the end product: a freely downloadable program that you can run on your computer or even a complete computer operating system in the case of Linux.  But to open source developers, open source is about more than this ‘free beer’ model of sharing software.  Open source software is shared at the source code level allowing people to examine the way it works, suggest changes to fix bugs, enhance it or even to modify it for their own purposes.  Getting the most from sharing source code requires more than just sharing an executable or a zip file of the finished product, open source developers need to open up their source code control systems too.

For years there have been services that provide a cloud-based alternative to  hosting your own source code.  The SourceForge system enjoyed many years of dominance but more recently it’s advertising sponsored model has seen it fall out of favour.

Most new projects are now created on a service called GitHub, which promises  free hosting of open source projects on a service funded by paying customers who are developing projects privately on the same platform.  The success of GitHub has been phenomenal – Google closed down its own rival service (Google Code) largely because of GitHub’s success.  In fact, GitHub is rapidly becoming a ‘unicorn’ with all the associated growing pains.  GitHub makes it easy to collaborate on projects too with its issue tracking system and user friendly tools for proposing changes (known as ‘pull requests’).

With GitHub as the de facto place to publish and share source code, it makes sense for Questionmark to use it to complement our Open Assessment Platform.  We have published source code illustrating how to use our APIs for many years and even publish the complete source to some of our connectors.  Putting new projects on GitHub means providing sample code in the most transparent and developer-friendly way possible.

Questionmark’s GitHub page lists all the projects we own.  For example, when we first brought out our OData APIs we published the sample reportlet code in the OData Reportlet Samples project.  You can experiment with these same examples running live in our website’s developer pages.

Recently we’ve gone a step further in opening up our assessment platform.  We’ve started publishing our API documentation via GitHub too!  Using a new feature of the GitHub platform we’re able to publish the documentation directly from the source control system itself.  That means you always get access to the latest documentation.

Opening up our API documentation in this way makes it easier for developers to engage with our platform.  Why not check out the documentation project.  If you’re already a GitHub user you could ‘watch’ it to get notified when we make changes.  You can even submit issues or send us ‘pull requests’ if you have suggestions for improvement.

With GitHub as the de facto place to publish and share source code, it makes sense for Questionmark to use it to complement our Open Assessment Platform.  We have published source code illustrating how to use our APIs for many years and even publish the complete source to some of our connectors.  Publishing this source code helps our customers and partners by providing working examples of how to integrate with our platform as well as providing complete transparency for our connectors allowing customers to audit the code before they run it on their own systems.  Putting new projects on GitHub means providing sample code in the most transparent and developer-friendly way possible.

Acronyms, Abbreviations and APIs

Steve Lay HeadshotPosted by Steve Lay

As Questionmark’s integrations product owner, it is all too easy to speak in acronyms and abbreviations. Of course, with the advent of modern day ‘text-speak,’ acronyms are part of everyday speech. But that doesn’t mean everyone knows what they mean. David Cameron, the British prime minister, was caught out by the everyday ‘LOL’ when it was revealed during a recent public inquiry that he’d used it thinking it meant ‘lots of love’.

In the technical arena things are not so simple. Even spelling out an acronym like SOAP (which stands for Simple Object Access Protocol) doesn’t necessarily make the meaning any clearer. In this post, I’m going to do my best to explain the meanings of some of the key acronyms and abbreviations you are likely to hear talked about in relation to Questionmark’s Open Assessment Platform.

API

At a recent presentation (on Extending the Platform), while I was talking about ways of integrating with Questionmark technologies, I asked the audience how many people knew what ‘API’ stood for. The response prompted me to write this blog article!

The term, API, is used so often that it is easy to forget that it is not widely known outside of the computing world.

API stands for Application Programming Interface. In this case the ‘application’ refers to some external software that provides functionality beyond that which is available in the core platform. For example, it could be a custom registration application that collects information in a special way that makes it possible to automatically create a user and schedule them to a specified assessment.

The API is the information that the programmer needs to write this registration application. ‘Interface’ refers to the join between the external software and the platform it is extending. (Our own APIs are documented on the Questionmark website and can be reached directly from developer.questionmark.com.)

APIs and Standards

APIs often refer to technical standards. Using standards helps the designer of an API focus on the things that are unique to the platform concerned without having to go into too much incidental detail. Using a common standard also helps programmers develop applications more quickly. Pre-written code that implements the underlying standard will often be available for programmers to use.

To use a physical analogy, some companies will ask you to send them a self-addressed stamped envelope when requesting information from them. The company doesn’t need to explain what an envelope is, what a stamp is and what they mean by an address! These terms act a bit like technical standards for the physical world. The company can simply ask for one because they know you understand this request. They can focus their attention on describing their services, the types of requests they can respond to and the information they will send you in return.

QMWISe

QMWISe stands for Questionmark Web Integration Services Environment. This API allows programmers to exchange information with Questionmark OnDemand software-as-a-service or Questionmark Perception on-premise software. QMWISe is based on an existing standard called SOAP. (see above)

SOAP defines a common structure used for sending and receiving messages; it even defines the concept of a virtual ‘envelope’. Referring to the SOAP standard allows us to focus on the contents of the messages being exchanged such as creating participants, creating schedules, fetching results and so on.

REST

REST stands for REpresentational State Transfer and must qualify as one of the more obscure acronyms! In practice, REST represents something of a back-to-basics approach to APIs when contrasted with those based on SOAP. It is not, in itself, a standard but merely a set of stylistic guidelines for API designers defined by an academic paper written by Roy Fielding, a co-author of the HTTP standard (see below).

As a result, APIs are sometimes described as ‘RESTful’, meaning they adhere to the basic principles defined by REST. These days, publicly exposed APIs are more likely to be RESTful than SOAP-based. Central to the idea of a RESTful API is that the things your API deals with are identified by a URL (Uniform Resource Locator), the web’s equivalent of an address. In our case, that would mean that each participant, schedule, result, etc. would be identified by its own URL.

HTTP

RESTful APIs draw heavily on HTTP. HTTP stands for HyperText Transfer Protocol. It was invented by Tim Berners-Lee and forms one of the key inventions that underpin the web as we know it. Although conceived as a way of publishing HyperText documents (i.e., web pages), the underlying protocol is really just a way of sending messages. It defines the virtual envelope into which these messages are placed. HTTP is familiar as the prefix to most URLs.

OData

Finally this brings me to OData. OData just stands for Open Data. This standard makes it much easier to publish RESTful APIs. I recently OData in the post, What is Odata, and why is it important?

Although arguably simpler than SOAP, OData provides an even more powerful platform for defining APIs. For some applications, OData itself is enough, and tools can be integrated with no additional programming at all. The PowerPivot plugin for Microsoft Excel is a good example. Using Excel you can extract and analyse data using the Questionmark Results API (itself built on OData) without any Questionmark-specific programming at all.

For more about OData, check out this presentation on Slideshare.

Using Questionmark to conduct a Performance Based Certification

John Kleeman HeadshotHow do you measure people’s practical skills, for example their ability to conduct electrical or electronic work? Is it possible to have someone use test equipment such as a hardware device or simulator and have this controlled by Questionmark technology?

Dave Fricton and Carolyn Parise The answer to this question is “yes”. I’d like to share an interesting story by Questionmark customer SpaceTEC, whose very inventive approach is making this happen.

SpaceTEC is the National Science Foundation, National Resource Center for Aerospace Technical Education. I’m grateful to DSpaceTec logoave Fricton and Carolyn Parise of SpaceTEC (pictured right) for presenting about this at the Questionmark Users Conference in San Antonio, Texas and for helping me write this blog article.

SpaceTEC, under its sister organization CertTEC, created an electronics certification that is offered to the electrical/electronics industry. Staff members there deliver knowledge based assessments with Questionmark and also deliver a practical exam where someone has to actually perform some electrical work, for instance finding a fault or make measurements. To do this, they use electrical test consoles and proprietary card sets like those shown below.

Model 130E Test console from NIDA                                 Electronic console

Traditionally the practical exam has been delivered manually by an examiner working with the candidate on the test equipment. But this is costly and difficult to organize nationwide as exams take 3 to 4 hours each and examiners need specialized training.

The innovation by SpaceTEC is that they have inserted HTML code inside Questionmark questions to control the test equipment. They drive the test equipment from within Questionmark software, making it no longer necessary for a trained examiner to run the practical test. They still have a proctor in the room to check the integrity of the process, but this is much easier to organize.

Here is a simple example of the kind of question they deliver. The candidate clicks on the Insert Fault button and this sets up the console with the appropriate configuration. Then the candidate measures the resistance on the console and types in their answer to Questionmark, which records and scores the question.

What is the resistance measurement in Kohms between TP1 and TP2?

If you want to know what happens behind the scenes, it’s very simple. The console (from NIDA Corporation) has an ActiveX driver which can be called by HTML code. It is necessary to install the driver on the candidate PC and then plug the console into the PC, but these are simple tasks. The call to the ActiveX driver is then encapsulated inside the Questionmark question wording.

Screenshot of HTML used

This is an example of Questionmark’s Open Assessment Platform, where you can connect Questionmark into other systems to get the solution you need. To quote Dave Fricton in his conference session: “The beauty of using Questionmark is you can do it all yourself”

Do you deliver any practical or performance exams using test equipment? If so, you might be able to follow the same route that SpaceTEC have gone, and link these up to Questionmark for easier administration. SpaceTEC are showing how performance and practical tests can be run in Questionmark, as well as knowledge assessments.

Need Customized Reports? Try Out Our OData API

Sample Attempt Distribution

Sample Attempt Distribution Reportlet

Joan Phaup 2013 (3)Posted by Joan Phaup

The standard reports and analytics that our customers use to evaluate assessment results meet a great many needs, but some occasions call for customized reports. The Questionmark OData API makes it possible to access data securely and create dynamic assessment reports using third-party business intelligence tools.

Once these reports are set up, they provide a flow of data, updating the results as new data becomes available. OData also makes it possible to cross-reference your assessment data with another data source  to get a fuller picture of what’s happening in your organization.

My recent interview with Austin Fossey goes into more detail about this, but you explore this idea yourself thanks to the OData tutorials and dashboards  on Questionmark’s Open Assessment Platform for developers.

The site provides example open source code to show how your organization could provide reportlets displaying key performance indicators from many types of assessment data.  The examples demonstrate these sample reportlets:

  • Attempt Distribution
  • PreTest PostTest
  • Score Correlation
  • Distribution

Questionmark OnDemand customers can plug in their own data to create their own reportlets, and developers can use tutorials to get detailed instructions about connecting to OData, retrieving data and creating charts.

You can also learn a lot about the power of OData at the upcoming Questionmark Users Conference in San Antonio March 4 – 7, so we hope you’ll join us there!

Using OData for dynamic, customized reporting: Austin Fossey Q&A

Joan Phaup 2013 (3)Posted by Joan Phaup

We’ll be exploring the power of the Open Data Protocol (OData) and its significance for assessment and measurement professionals during the Questionmark 2014 Users Conference in San Antonio March 4 – 7.

Austin Fossey, our reporting and analytics manager, will explain the ins and outs of using the Questionmark OData API, which makes it possible to access assessment results freely and use third-party tools to create dynamic, customized reports. Participants in a breakout session about the OData API, led by Austin along with Steve Lay, will have the opportunity to try it out for themselves.

Austin Fossey-42

Austin Fossey

I got some details about all this from Austin the other day:

What’s the value of learning about the OData API?

The OData API gives you access to raw data. It’s an option for accessing data from your assessment results warehouse without having to know how to program, query databases or even host the database yourself. By having access to those data, you are not limited to the reports Questionmark provides: You can do data merges and create your own custom reports.

OData is really good for targeting specific pieces of info people want. The biggest plus is that it doesn’t just provide data access. It provides a flow of data. If you know the data you need and you want to set up a report, a spreadsheet, or just have it in the web browser, you can get those results updated as new data become available. This flow of data is what makes OData reports truly dynamic, and this is what distinguishes OData reports from reports that are built from manually generated data exports.

What third-party tools can people use with the OData API?

Lots! Key applications include Microsoft Excel PowerPivot, Tableau, the Sesame Data Browser, SAP Business Objects and Logi Analytics, but there are plenty to choose from. People can also do their own programming if they prefer.  The Odata.org website includes a helpful listing of the OData ecosystem, which includes applications that generate and consume OData feeds.

Can you share some examples of custom reports that people can create with OData?

We have some examples of OData reportlets on our Open Assessment Platform website for developers, which also includes some tutorials. I’ve blogged about using the OData API to create a response matrix and to create a frequency table of item keys in Microsoft PowerPivot for Excel. There are so many different ways to use this!

What about merging data from assessments with data from other sources? What are some scenarios for doing that?

It could be any research where you want to cross-reference your assessment data with another data source. If you have another data set and were able to identify participants – say an HR database showing the coursework people have done – you could compare that with their test results to their course activity Reports don’t necessarily have to be about test scores. They can be about items and answer choices – anything you want.

Tell me about the hands-on element of this breakout session.

We will be working through a fairly simple example using Microsoft PowerPivot for Excel  in order to cement the concepts of using OData. We’re encouraging people to bring their laptops with Excel and the PowerPivot add-in already installed. If they don’t have that, they can either work with someone else or watch the exercise onscreen. We will provide a handout explaining everything so they can try this when they are back at work.

What do you want people to take away from this breakout?

We want to make sure people know how to construct an OData URL, that they understand the possibilities of using OData but also the limitations. It won’t be a panacea for everything. We want to be sure they know they have another tool in their tool box to answer the research questions or business questions they encounter day to day.

Our conference keynote speaker, Learning Strategist Bryan Chapman, will share insights about OData and examples of how organizations are using it during his presentation on Transforming Data into Meaning and Action.

Click here to see the complete conference program. And don’t forget to sign up by January 30th if you want to save $100 on your registration.

What is OData, and why is it important?

Steve Lay HeadshotPosted by Steve Lay

At the recent Questionmark Users Conference I teamed up with Howard Eisenberg, our Director of Solution Services, to talk about OData. Our session included some exciting demonstrations of our new OData API for Analytics. But what is OData and why is it important?

OData is a standard for providing access to data over the internet. It has been developed by Microsoft as an open specification. To help demonstrate its open approach, Microsoft is now working with OASIS to create a more formal standard. OASIS stands for the Organization for the Advancement of Structured Information Standards; it provides the software industry with a way to create standards using open and transparent procedures. OASIS has published a wide range of standards, particularly in the areas of document formats and web service protocols — for example, the OpenDocument formats used by the OpenOffice application suite.

Why OData?

Questionmark’s Open Assessment Platform already includes a set of web-service APIs (application programming interfaces). We call them QMWISe and they are ideal for programmers who are integrating server-based applications. With one QMWISe request you can trigger a series of actions typical of a number of common use cases. There are, inevitably, times when you need more control over your integration, though, and that is where OData comes in.

Unlike QMWISe, OData provides access to just the data you want. It has scalability built right in to the protocol. Using the conventions of OData, you can make highly specific requests to get a single data item or you can use the feature of linked-data to quickly uncover relationships.

OData works just like the web: each record returned by an OData request contains links to other related records in exactly the same way as web pages contain hyperlinks to other web pages. Want to know about all the results for a specific assessment? It is easy with OData, just follow the results link in the assessment’s record.

OData is also based on pre-existing internet protocols, which means that web developers can use it in their applications with a much easier learning curve. In fact, if a tool already supports RSS/Atom you can probably start accessing OData-feeds right away!

OData Ecosystem

As we build our support for the OData protocol, we join a growing community. OData makes sense as the starting point for any data-rich standard. Last week I was at CETIS 2013, where there was already talk of other standards organizations in the e-Learning community adopting OData as a way of standardizing the way they share information.