Practice versus perfection – 2014 Users Conference

Austin Fossey-42 Posted by Austin Fossey

The Questionmark team has just returned from the 2014 Users Conference, where we had a wonderful time showing off our latest work, discussing assessment strategies with our customers, and learning from each other in a great selection of seminars put on by both Questionmark staff and our clients.

At this year’s conference, I field tested two new presentations: Understanding Assessment Results and Principles of Psychometrics and Measurement Design. I got some great feedback from attendees so I can fine-tune them for the future, but these topics also started a lot of interesting conversations about what we as test developers would like to be doing and what we end up doing in practice.

A recurring theme of these conversations was that people felt there were occasionally aspects of their instruments that could be improved, especially in terms of capturing evidence for a measurement or supporting the validity of the results. In some cases they had an idea of what they wanted to improve, but they either did not know the test development methods they needed to apply, or they did not know how to convince their stakeholders and managers of the importance of specific initiatives. The concept of validity came up several times in these conversations—something we have touched on previously on this blog.

The ideals and realities of the assessment industry do not always align. For example, we may wish to do a construct validity study or an Angoff cut score meeting, but we may lack the resources, time, or stakeholder buy-in to engage in these activities.

I recognize how discouraging this can be for people who constantly want to improve the validity of their inferences, but I am excited to see so many people thinking critically about their assessment designs and searching for areas of improvement. Even if we cannot always implement every research study we are interested in, understanding the principles and best practices of good assessment design and interpretation can still guide our everyday work and help us to avoid invalid results. This blog is a good place to explore some of these principles, and so are Questionmark white papers and our Learning Café videos.

I look forward to continuing to work with (and learn from) our great client base throughout 2014 as we continue to advance our products. A special thanks to our attendees and presenters who joined us at the 2014 conference!

Setting Your Data Free – 2014 Users Conference

Austin Fossey-42Posted by Austin Fossey

The Questionmark Product Team is off to the 2014 Users Conference! We had a great time last night at the opening reception and are ready now to launch into the conference program.

A major theme this year is “setting your data free!” — so I wanted to give you a little taste of how this theme relates to my presentations on reporting and analytics.

As you know from my previous posts, we have implemented the OData API, which connects your raw assessment data (the same data driving Questionmark Analytics) to a whole ecosystem of business intelligence tools, custom dashboards, statistical packages, and even common desktop applications like Excel and web browsers. At this year’s conference, we will talk about how OData can be a tool for freeing those data for users of all types. Be sure to check out my OData session, where we will be running through examples using Excel with the PowerPivot add-in.

night riverBut, when we free our data, we want to make sure we are putting good, quality, meaningful data out there for our stakeholders so that they make valid inferences about the participants and the assessments. I will be doing two presentations related to this topic. In one, we will talk about understanding assessment results, with a focus on the classical test theory model and its applications for evaluation assessment quality with item statistics. In the second presentation, we will talk about principles of psychometrics and measurement design, where we will discuss validity studies and how principled test development frameworks like evidence-centered design can help us build better assessments that produce actionable data.

I’m pleased to see everyone in San Antonio and expect to be talking a lot about how we can set data free to make a powerful impact for stakeholders!

Create high-quality assessments: Join a March 4 workshop

Joan Phaup 2013 (3)

Posted by Joan Phaup

There will be a whole lot of learning going on in San Antonio March 4 during three workshops preceding the Questionmark 2014 Users Conference.

These sessions cover a broad range of experience levels — from people who are just beginning to use Questionmark technologies to those who want to understand best practices in test development and item writing.

Rick Ault

Rick Ault

Questionmark Boot Camp: Basic Training for Beginners (9 a.m. – 4 p.m.)

Questionmark Trainer Rick Ault will lead this hands-on workshop, which begins with a broad introduction to the Questionmark platform and then becomes an interactive, hands-on practice session. Bring your own laptop to get some firsthand experience creating and scheduling assessments. Participants will also get acquainted with reports and analytics.

Melissa Fein web

Dr. Melissa Fein

Test Development Fundamentals (9 a.m. – 12 p.m.)

Whether you are involved in workplace testing, training program evaluation, certification & certificate program development, or academic testing, an understanding of criterion-referenced test development will strengthen your testing program. Dr. Melissa Fein, author of Test Development Fundamentals for Certification and Evaluation, leads this morning workshop, which will help participants judge test quality, set mastery cutoff points, and improve test quality.


Mary Lorenz

The Art and Craft of Item Writing (1 p.m. – 4 p.m.)

Writing high-quality multiple-choice questions can present many challenges and pitfalls. Longtime educator and test author Mary Lorenz will coach workshop participants through the process of constructing well-written items that measure given objectives. Bring items of your own and sharpen them up during this interactive afternoon session.


Choose between the full-day workshop and one or both of the half-day workshops.

Conference attendees qualify for special workshop registration rates, and there’s a discount for attending both half-day sessions.

Click here for details and registration.




Early-birds: Check out the conference program and register by tomorrow

Joan Phaup 2013 (3)Posted by Joan Phaup

You can still get a $100 early-bird registration discount for the Questionmark 2014 Users Conference if you register by tomorrow (Thursday, January 30th).

The program for March 4 – 7 includes a keynote by Learning Strategist Bryan Chapman on the power of open data, which will be a hot topic throughout this gathering at the Grand Hyatt San Antonio.

You can click here for program details, but here’s the line-up:

Optional Pre-Conference Workshopsbreakout 2013

  • ​Questionmark Boot Camp: Basic Training for Beginners  ​(full day)
  • ​Test Development Fundamentals  (half day)
  • ​The Art & Craft of Item Writing   ​(half day)

Case Studies

  • Using Questionmark to Conduct Performance Based Certifications —  SpaceTEC®
  • Better Outcomes Make the Outcome Better! — US Marine Corps University
  • ​Generating and Sending Custom Completion Certificates —  The Aurelius Group
  • ​Leveraging Questionmark’s Survey Capabilities with a Multi-system Model – Verizon
  • ​Importing Questions into Questionmark Live on a Tri-Military Service Training Campus Medical Education & Training Campus
  • How Can a Randomly Designed Test be Fair to All? US Coast Guard Training Center

Best Practices doug teaching 2013

  • ​Principles of Psychometrics and Measurement Design
  • 7 Reasons to Use Online Assessments for Compliance
  • ​Reporting and Analytics: Understanding Assessment Results
  • ​Making it Real: Building Simulations Into Your Quizzes and Tests
  • ​Practical Lessons from Psychology Research to Improve Your Assessments
  • Item Writing Techniques for Surveys, Quizzes and Tests

Questionmark Features & Functions

  • Introduction to Questionmark for Beginners
  • BYOL: Item and Topic Authoring
  • BYOL: Collaborative Assessment Authoring
  • Integrating with Questionmark’s Open Assessment Platform
  • Using Questionmark’s OData API for Analytics (BYOL)
  • Successfully Deploying Questionmark Perception
  • Customizing the Participant Interface


  • Testing What We Teach: How can we elevate our effectiveness without additional time or resources?
  • Testing is Changing: Practical and Secure Assessment in the 21st Century

Future Solutions Focus Groups

  • Open Assessment Platform and Analytics
  • Authoring and Delivery

Special Interest Group Meetings

  • Military/Defense US DOD and Homeland Security
  • Utilities/Energy Generation and Distribution
  • Higher Education
  • Universities

Drop-in Demos of new Questionmark features and capabilities

Tech Central: Drop-in meetings with Questionmark technicians


Register for the conference by tomorrow, Thursday, January 31st, to get the early-bird discount.



Using OData for dynamic, customized reporting: Austin Fossey Q&A

Joan Phaup 2013 (3)Posted by Joan Phaup

We’ll be exploring the power of the Open Data Protocol (OData) and its significance for assessment and measurement professionals during the Questionmark 2014 Users Conference in San Antonio March 4 – 7.

Austin Fossey, our reporting and analytics manager, will explain the ins and outs of using the Questionmark OData API, which makes it possible to access assessment results freely and use third-party tools to create dynamic, customized reports. Participants in a breakout session about the OData API, led by Austin along with Steve Lay, will have the opportunity to try it out for themselves.

Austin Fossey-42

Austin Fossey

I got some details about all this from Austin the other day:

What’s the value of learning about the OData API?

The OData API gives you access to raw data. It’s an option for accessing data from your assessment results warehouse without having to know how to program, query databases or even host the database yourself. By having access to those data, you are not limited to the reports Questionmark provides: You can do data merges and create your own custom reports.

OData is really good for targeting specific pieces of info people want. The biggest plus is that it doesn’t just provide data access. It provides a flow of data. If you know the data you need and you want to set up a report, a spreadsheet, or just have it in the web browser, you can get those results updated as new data become available. This flow of data is what makes OData reports truly dynamic, and this is what distinguishes OData reports from reports that are built from manually generated data exports.

What third-party tools can people use with the OData API?

Lots! Key applications include Microsoft Excel PowerPivot, Tableau, the Sesame Data Browser, SAP Business Objects and Logi Analytics, but there are plenty to choose from. People can also do their own programming if they prefer.  The website includes a helpful listing of the OData ecosystem, which includes applications that generate and consume OData feeds.

Can you share some examples of custom reports that people can create with OData?

We have some examples of OData reportlets on our Open Assessment Platform website for developers, which also includes some tutorials. I’ve blogged about using the OData API to create a response matrix and to create a frequency table of item keys in Microsoft PowerPivot for Excel. There are so many different ways to use this!

What about merging data from assessments with data from other sources? What are some scenarios for doing that?

It could be any research where you want to cross-reference your assessment data with another data source. If you have another data set and were able to identify participants – say an HR database showing the coursework people have done – you could compare that with their test results to their course activity Reports don’t necessarily have to be about test scores. They can be about items and answer choices – anything you want.

Tell me about the hands-on element of this breakout session.

We will be working through a fairly simple example using Microsoft PowerPivot for Excel  in order to cement the concepts of using OData. We’re encouraging people to bring their laptops with Excel and the PowerPivot add-in already installed. If they don’t have that, they can either work with someone else or watch the exercise onscreen. We will provide a handout explaining everything so they can try this when they are back at work.

What do you want people to take away from this breakout?

We want to make sure people know how to construct an OData URL, that they understand the possibilities of using OData but also the limitations. It won’t be a panacea for everything. We want to be sure they know they have another tool in their tool box to answer the research questions or business questions they encounter day to day.

Our conference keynote speaker, Learning Strategist Bryan Chapman, will share insights about OData and examples of how organizations are using it during his presentation on Transforming Data into Meaning and Action.

Click here to see the complete conference program. And don’t forget to sign up by January 30th if you want to save $100 on your registration.

Keep those conference comments coming through December 1

Joan Phaup 2013 (3)

Posted by Joan Phaup

Thinking of Thanksgiving, something we are  always grateful for here at Questionmark is, of course, our customers!

I’d like to thank people who have posted comments on our Facebook page as part of our 2014 Users Conference sweepstakes.

Those who “like” our page and post comments on the conference banner there are being entered into a random drawing for a free conference registration plus a food service gift certificate from the Grand Hyatt San Antonio.

If you have not done this yet, there’s still time! The sweepstakes ends Sunday, December 1, so take a moment to tell us why you’d like to attend the conference. We’ll put your name in the hat along with all the others. And if you’ve already registered, we’ll refund your fee.

Here are just a few of the answers we’ve received to the question, “Why would you like to attend the Questionmark 2014 Users Conference?”

  • “Excellent learning opportunities…and tacos.”Facebook Sweepstakes Final Banner 10-31-13
  • “I met so many great people at last year’s conference in Baltimore that were using Questionmark in so many different ways to support training. Good opportunity to pick everyone’s brains!”
  • ” I have never attended before and Questionmark has become essential in my day-to-day. So I’d like face-to-face interaction with other users and the Questionmark staff.”
  • “Would enjoy seeing what’s new with Questionmark and what is planned for the future.”
  • “We are getting ready to implement testing through Questionmark, so conference attendance would be a great opportunity to learn more about the software and network!”
  • “I would like to hear stories from other users as well as meet more of the Questionmark staff, who have always been generous in their assistance.”

Then there was the entrant who raved about everything from Tech Central and breakout presentations to the fun of meeting people with common  interests. then simply put into words the wish of everyone who is written in so far: “PICK ME!!!”

The winner’s name will be drawn at random on December 2nd. If you’re not on Facebook or have questions about the sweepstakes rules, click here.

So, what’s your answer to the question? Like us on Facebook if you haven’t already — and click on the conference sweepstakes banner there to tell us why you’d like to attend the conference. The deadline for entries is Sunday, December 1 — so go to our Facebook page soon — to “Like,” “Click,” and “Comment.”