Conference Wrap Up: Tips, Advice & Pictures from Napa

Julie Delazyn HeadshotPosted by Julie Delazyn

flickrThe Questionmark Users Conference is the most important learning event of the year. With over a dozen sessions to attend and topics ranging from penetration testing to measuring and understanding your assessment results, there is so much knowledge packed within three days.

Assessment security was an important topic at the Conference in Napa. Questionmark Chairman John Kleeman took to the blog last week to lay out some security advice he heard from attendees. You can check out his blog post, Assessment Security: 5 Tips from Napa, to learn more.

Frequent blog contributor, Psychometrician and Reporting and Analytics Manager, Austin Fossey, presented a number of  sessions at the conference this year. According to Austin, “regardless of their individual roles or organizations’ goals, Questionmark users are first and foremost measurement professionals.” Impressed by our customers’ commitment to look for ways to always improve their measurements, validity, and impact for stakeholders, Austin wrote a blog post highlighting some of the example that struck him. You can read more about the stories he heard from our customers about their assessment programs in his blog post: 2015 Users Conference – A Gathering of Measurement Professionals.

If you did not have a chance to attend the Conference in Napa, there is always next year! Look out for dates and a special location announcement on the blog. For those of you who attended and would like to relive some of the special moments spent in Napa, you can check out the pictures now on our Flickr page. The photos highlight moments from the conference as well as from our special evening event at Markham Winery.

conf goers banner

Interact with your data: Looking forward to Napa

Steve Lay HeadshotPosted by Steve Lay

It’s almost time for the Questionmark Users Conference, which this year is being held in Napa, California. As usual there’s plenty on the program for delegates interested in integration matters!

At last year’s conference we talked a lot about OData for Analytics, (which I have also written about here: What is OData, and why is it important? ). OData is a data standard originally created by Microsoft but now firmly embedded in the open standards community through a technical group at OASIS. OASIS have taken on further development, resulting in the publication of the most recent version, OData 4.

This year we’ve built on our earlier work with the Results OData API to extend our adoption of OData to our delivery database, but there’s a difference. Whereas the Results OData API provides access to data, the data exposed from our delivery system supports read and write actions, allowing third-party integrations to interact with your data during the delivery process.

Why would you want to do that?

Some assessment delivery processes involve actions that take place outside the Questionmark system. The most obvious example is essay grading. Although the rubrics (the rules for scoring) are encoded in the Questionmark database, it takes a human being outside the system to follow those rules and to assign marks to the participant. We already have a simple scoring tool built directly in to Enterprise Manager but for more complex scoring scenarios you’ll want to integrate with external marking tools.

The new Delivery OData API provides access to the data you need, allowing you to read a participant’s answers and write back the scores using a simple Unscored -> Saved -> Scored workflow. When the result is placed in the final status, the participant’s result is updated and will appear with the updated scores in future reports.

I’ll be teaming up with Austin Fossey, our product owner for reporting, and Howard Eisenberg, our head of Solution Services, to talk at the conference about Extending Your Platform, during which we’ll be covering these topics. I’m also delighted that colleagues from Rio Salado College will also be talking about their own scoring tool that is built right on top of the Delivery OData API.

I look forward to meeting you in Napa but if you can’t make it this year, don’t worry, some of the sessions will be live-streamed. Click here to register so that we can send you your login info and directions. And you can always follow along with social media by following and tweeting with @Questionmark.

Can’t travel to Napa? Be there virtually!

Much as we would like to see all of our customers at the annual Questionmark Users Conference, we know it is not always possible to get there.grapes gray

Here’s what to do if you can’t join us in Napa Valley March 10 – 13: catch selected conference sessions via webcasts.

Register soon to experience these conference highlights online:

  • Opening General Session
  • Introduction to Item Development in Large-Scale Test Development
  • Hacking yourself first: Protecting your assessment data with penetration testing
  • Conference Keynote
  • Overview of Setting Performance Standards: Making the cognitive leap from scores to interpretations
  • Writing Performance-Based Test Items

Sign up for these complimentary conference webcasts (for customers only)

Experiencing the complete conference offers by far the best way to learn about assessment-related best practices and about how to make the most of Questionmark technologies.

There’s still time to register for the full conference, but if you can’t make it this time around, we certainly hope you will take advantage of these webcasts.

2015_uc_napa_banner1

conf goers banner

Standard Setting: A Keystone to Legal Defensibility

Austin Fossey-42Since the last Questionmark Users Conference, I have heard several clients discuss new measures at their companies requiring them to provide evidence of the legal defensibility of their assessment. Legal defensibility and validity are closely intertwined, but they are not synonymous. An assessment can be legally defensible, yet still have flaws that impact its validity. The distinction between the two is often the difference between how you developed the instrument versus how well you developed the instrument.

Regardless of whether you are concerned with legal defensibility or validity, careful attention should be paid to the evaluative component of your assessment program. What if  someone asks, “What does this score mean?” How do you answer? How do you justify your response? The answers to these questions impact how your stakeholders will interpret and use the results, and this may have consequences for your participants. Many factors go into supporting the legal defensibility and validity of assessment results, but one could argue that the keystone is the standard-setting process.

Standard setting is the process of dividing score scales so that scores can be interpreted and actioned (AERA, APA, NCME, 2014). The dividing points between sections of the scales  are called “cut scores,” and in criterion-referenced assessment, they typically correspond to performance levels that are defined a priori. These cut scores and their corresponding performance levels help test users make the cognitive leap from a participant’s response pattern to what can be a complex inference about the participant’s knowledge, skills, and abilities.

In their chapter in Educational Measurement (4th Ed.), Hambleton and Pitoniak explain that standard-setting studies need to consider many factors, and that they also can have major implications for participants and test users. For this reason, standard-setting studies are often rigorous, well-documented projects.

At this year’s Questionmark Users Conference, I will be delivering a session that introduces the basics of standard setting. We will discuss standard-setting methods for criterion- referenced and norm-referenced assessments, and we will touch on methods used in both large-scale assessments and in classroom settings. This will be a useful session for anyone who is working on documenting the legal defensibility of their assessment program or who is planning their first standard-setting study and wants to learn about different methods that are available. Participants are encouraged to bring their own questions and stories to share with the group.

Register today for the full conference, but if you cannot make it, make sure to catch the live webcast!

Napa Conference Overview & Special Live Webcast

Headshot JuliePosted by Julie Delazyn

The Questionmark Users Conference in Napa Valley March 10-13 is officially less than a month away!2015-napa-01

We are looking forward to seeing you there for three full days of learning and special events.

Experiencing the complete conference offers by far the best way to learn about assessment-related best practices and about how to make the most of Questionmark technologies. Participants will hear customer case studies and learn about the unique ways in which other organizations are using assessments. They will also weigh in on the product road map, connect with Questionmark staff and network with their peers while enjoying California’s beautiful wine country.

But here’s a great opportunity for Questionmark users who can’t travel to the conference this year: live webcasts of selected conference sessions.

These select sessions will be broadcast at no charge March 11-12:

  • Opening General Session
  • Introduction to Item Development in Large-Scale Test Development
  • Hacking yourself first: Protecting your assessment data with penetration testing
  • Conference Keynotecollage
  • Overview of Setting Performance Standards: Making the cognitive leap from scores to interpretations
  • Writing Performance-Based Test Items

Make sure to sign up for conference webcasts

Aside from getting to take advantage of beautiful California wine country, there’s much more in store for conference participants:

Register today for the full conference, but if you cannot make it, make sure to catch the live webcast

Q&A: High-stakes online tests for nurses

Headshot JuliePosted by Julie Delazyn

I spoke recently with Leanne Furby, Director of Testing Services at the National League for Nursing (NLN), about her case study presentation at the Questionmark 2015 Users Conference in Napa Valley March 10-13.

Leanne’s presentation, Transitioning 70 Years of High-Stakes Testing to Questionmark, explains NLN’s switch from a proprietary computer- and paper-based test delivery engine to Questionmark OnDemand for securely delivering standardized exams worldwide. I’m happy to share a snippet from of our conversation:

Tell me about the NLN

The NLN is a national organization for faculty nurses and leaders in nurse education. We offer faculty development, networking opportunities, testing services, nursing research grants and public policy initiatives to more than 26,000 members.

Why did you switch to Questionmark?

Our main concern was delivering our tests and exams to a variety of different devices. We wanted our students to be able to take a test on a tablet or take a quiz on their own mobile devices, and this wasn’t something we could do with our proprietary test delivery engine.

Our second major reason to go with Questionmark was the Customized Assessment Reports and the analytics tools. Before making the switch, we were having to create reports and analyze results manually. It took time and resources. Now this is all integrated in Questionmark.

How do you use Questionmark assessments?

We have 90 different exam lines and deliver approximately 75,000 to 100,000 secure exams a year, both nationally and internationally, in multiple languages. The NLN partnered with Questionmark in 2014 to transition the delivery of these exams through a custom-built portal. Questionmark is now NLN’s turnkey solution—from item banking and test development with SMEs all over the world to inventory control, test delivery and analytics.

This transition has had a positive outcomes for both our organization and our customers. We have developed a new project management policy, procedures for system transition and documentation for training at all levels. This has transformed the way we develop, deliver and analyze exams and the way we collect data for business and education purposes.

What are you looking forward to at the conference?

I am most looking forward to the opportunity to speak to other users and product developers to learn tips, tricks and little secrets surrounding the product. It’s so important to speak to people who have experience and can share ways of utilizing the software in ways you hadn’t thought of.

Thank you Leanne for taking time out of your busy schedule to discuss your session with us!

***

You have the opportunity to save $100 on your own conference registration: Just sign up by January 29 to receive this special early-bird discount.