5 Things I Learned at the European Association of Test Publishers Conference Last Week

Posted by John Kleeman

I just attended the Association of Test Publisher’s European conference (EATP), held last week in Madrid, and wanted to share some of what I learned.

The Association of Test Publishers (ATP) is the trade association for the assessment industry and promotes good practice in assessment. Questionmark have been members for a long time and I am currently on their board of directors. The theme of the conference was “Transforming Assessments: Challenge. Collaborate. Inspire.”

Panel at European Association of Test Publishers

As well as seeing a bit of Madrid (I particularly enjoyed the beautiful Retiro Park), here are some things I learned at the conference. (These are all my personal opinions, not endorsed by Questionmark or the ATP).

1. Skills change. One area of discussion was skills change. Assessments are often used to measure skills, so as skills change, assessments change too. There were at least three strands of opinion. One is that workplace skills are changing rapidly – half of what you learn today will be out of date in five years, less if you work in technology. Another is that many important skills do not change at all – we need to collaborate with others, analyze information and show emotional resilience; these and other important skills were needed 50 years ago and will still be needed in 50 years’ time. And a third suggested by keynote speaker Lewis Garrad is that change is not new. Ever since the industrial revolution, there has been rapid change, and it’s still the case now. All of these are probably a little true!

2. Artificial Intelligence (AI). Many sessions at the conference covered AI. Of course, a lot of what gets called AI is in fact just clever marketing of smart computer algorithms. But nevertheless, machine learning and other things which might genuinely be AI are definitely on the rise and will be a useful tool to make assessments better. The industry needs to be open and transparent in the use of AI. And in particular, any use of AI to score people or identify anomalies that could indicate test cheating needs to be very well built to defend against the potential of bias.

3. Debate is a good way to learn. There were several debates at the conference, where experts debated issues such as performance testing, how to detect fraud and test privacy vs security, with the audience voting before and after. As the Ancient Greeks knew, this is a good format for learning, as you get to see the arguments on both sides presented with passion. I’d encourage others to use debates for learning.

4. Privacy and test security genuinely need balance. I participated in the privacy vs test security debate, and it’s clear that there is a genuine challenge balancing the privacy rights of individual test-takers and the needs of testing organizations to ensure results are valid and have integrity. There is no single right answer. Test-taker rights are not unlimited. And testing organizations cannot do absolutely anything they want to ensure security. The growing rise of privacy laws including the GDPR has brought discussion about this to the forefront as everyone seeks to give test-takers their mandated privacy rights whilst still being able to process data as needed to ensure test results have integrity. A way forward seems to be emerging where test-takers have privacy and yet testing organizations can assert legitimate interests to resist cheating.

5. Tests have to be useful as well as valid, reliable and fair. One of the highlights of the conference was a CEO panel, where Marten Roorda, CEO of ACT, Norihisa Wada, a senior executive at EduLab in Japan, Sangeet Chowfla, CEO of the Graduate Management Admission Council and Saul Nassé, CEO of Cambridge Assessment gave their views on how assessment was changing. I moderated this panel (see picture below) and it was great to hear these very smart thought leaders talk of the future.  There is widespread agreement that validity, reliability and fairness are key tenets for assessments , but also a reminder that we also need “efficacy” – i.e. that tests need to be useful for their purpose and valuable to those who use them.

There was a huge amount of other conference conversations including sessions on online proctoring, test translation, the update to the ISO 10667 standard, producing new guidelines on technology based assessment and much, much more.

I found it challenging, collaborative and inspiring and I hope this blog gives you a small flavor of the conference.

Final Early-Bird Discount Ends Next Week!

Register for Questionmark Conference 2017 by Wednesday, February 22, for a chance to take advantage of of our final early-bird discount.

Register now and save!

Here are some highlights from the agenda:
  • Assessments as Career Development Tools at Caterpillar Inc.
  • BYOL: Next Generation Assessment Authoring Using Demographics and Jump Blocks
  • Secure Authentication: Single sign-on for Questionmark
  • Staying Ahead of Evolving Security Threats
  • Setting a Fair, Defensible Cut (Passing) Scores

Don’t miss out on the opportunity to:

  • Maximize your Assessment Value
  • Harness the Power of your Results
  • Innovate with us on Future Assessment Technologies

Register now for early-bird savings
Book your room at the Inn and Spa at Loretto before Thursday, Feb. 16 to take advantage of our special rates

We look forward to seeing you in Santa Fe, New Mexico, March 21-24 for our most important learning event of the year!

Item Writing & Questionmark Boot Camp: Pre-Conference Workshops

Rick Ault, Questionmark Trainer

Julie ProfilePosted by Julie Delazyn

Planning for Questionmark Conference 2017 in Santa Fe, New Mexico, March 21-14 is well underway.

We have begun posting descriptions of breakout sessions and are pleased to announce two pre-conference workshops.

Both of these all-day sessions will take place on Tuesday, March 21, 2017:

assessments-2017Questionmark Boot Camp: Basic Training for Beginners

Do you want to get the most out of the Questionmark Users Conference even though you are just starting out with Questionmark?

Here’s what to do: Bring your laptop and learn directly from Questionmark expert Rick Ault.

Bring your laptop and get into gear with hands-on practice in creating questions, putting together an assessment, then scheduling it, taking it and seeing the results. Start off with some first-hand experience that will give you a firm footing for learning more at the conference.

jimparry_nov2016_xsmll

Jim Parry, Owner and Chief Executive Manager of Compass Consultants, LLC

features-functions-2017Advanced Test Item Writing Workshop: Learn how to test more than just knowledge

Writing test items is difficult, but trying to make them check more than knowledge is a huge challenge.

Join Certified Performance Technologist Jim Parry  — an expert user of Questionmark technologies — in this fast-paced, high-powered workshop, which will present a review of the basics of testing and provide hands-on practice to help you turn low complexity, knowledge-based test items into higher complexity, performance-based items following Bloom’s Taxonomy and Gagne’s Nine Events of Instruction.

Conference Registration Tuition:

  • You can save $200 by registering for the conference on or before January 18. You can sign up for a workshop at the same time or add in a workshop later. It’s up to you! Pre-conference workshop or bootcamp add-ons available during the registration process.

Calling all Speakers: Share your story in Santa Fe & become a VIP

Posted by Julie Delazyn

We have officially announced the Questionmark Conference 2017 March 21-14, and we look forward to seeing you at the beautiful Inn and Spa at Loretto for this important learning event.

In order to create a rich and varied conference program, we have opened a call for proposals and invite you to submit your idea for case study presentation or a peer discussion soon.

How do you know if you should submit a proposal? If you can answer yes to any of the questions below, we want to hear from you! Click here to find out more and download the proposal forms!

  • Your experience with Questionmark technologies will help others
  • You have found innovative ways to use online assessments
  • You can explain how you organized your assessment program and what you learned
  • You have gained a lot from previous conferences and want to contribute in 2015
  • You are using assessments to support organizational goals
  • You have a unique application of online or mobile assessments
  • You have integrated Questionmark with another system

We are accepting proposals from now until December 1, so consider what you’d like to contribute.

The Inn and Spa at Loretto

Aside from helping your fellow Questionmark users by sharing your story, please note that presenters will also get some perks:

Presenters and discussion leaders will receive some red carpet treatment — including a special dinner in their honor on Wednesday, March 21. And we award one 50% registration for each case study presentation.

Click here to start the process of becoming a speaker!

Even if you are not sure you’ll attend the conference, we would like to hear from you! And whether you plan to present or not, plan now to have the conference in your budget for 2017.

Conference Wrap Up: Tips, Advice & Pictures from Napa

Julie Delazyn HeadshotPosted by Julie Delazyn

flickrThe Questionmark Users Conference is the most important learning event of the year. With over a dozen sessions to attend and topics ranging from penetration testing to measuring and understanding your assessment results, there is so much knowledge packed within three days.

Assessment security was an important topic at the Conference in Napa. Questionmark Chairman John Kleeman took to the blog last week to lay out some security advice he heard from attendees. You can check out his blog post, Assessment Security: 5 Tips from Napa, to learn more.

Frequent blog contributor, Psychometrician and Reporting and Analytics Manager, Austin Fossey, presented a number of  sessions at the conference this year. According to Austin, “regardless of their individual roles or organizations’ goals, Questionmark users are first and foremost measurement professionals.” Impressed by our customers’ commitment to look for ways to always improve their measurements, validity, and impact for stakeholders, Austin wrote a blog post highlighting some of the example that struck him. You can read more about the stories he heard from our customers about their assessment programs in his blog post: 2015 Users Conference – A Gathering of Measurement Professionals.

If you did not have a chance to attend the Conference in Napa, there is always next year! Look out for dates and a special location announcement on the blog. For those of you who attended and would like to relive some of the special moments spent in Napa, you can check out the pictures now on our Flickr page. The photos highlight moments from the conference as well as from our special evening event at Markham Winery.

conf goers banner

Can’t travel to Napa? Be there virtually!

Much as we would like to see all of our customers at the annual Questionmark Users Conference, we know it is not always possible to get there.grapes gray

Here’s what to do if you can’t join us in Napa Valley March 10 – 13: catch selected conference sessions via webcasts.

Register soon to experience these conference highlights online:

  • Opening General Session
  • Introduction to Item Development in Large-Scale Test Development
  • Hacking yourself first: Protecting your assessment data with penetration testing
  • Conference Keynote
  • Overview of Setting Performance Standards: Making the cognitive leap from scores to interpretations
  • Writing Performance-Based Test Items

Sign up for these complimentary conference webcasts (for customers only)

Experiencing the complete conference offers by far the best way to learn about assessment-related best practices and about how to make the most of Questionmark technologies.

There’s still time to register for the full conference, but if you can’t make it this time around, we certainly hope you will take advantage of these webcasts.

2015_uc_napa_banner1

conf goers banner