Hurry! Early-Bird Registration Ends Today

Posted by Julie Delazyn

Today is the last day to save on your registration for Questionmark Conference 2017… And seats are going fast!

Take advantage of the early-bird discount and save $200, if you sign up by the end of the day!

Aside from some fantastic social events that are designed to assist you in networking with other assessment professionals, we have a jam packed agenda covering topics such as:

  • Avionics: A hands-on, performance-based certification using Questionmark at SpaceTEC
  • Assessments as Career Development Tools at Caterpillar Inc.
  • Explore the Advanced Editor in Next Gen Authoring
  • Why Use Performance-Based Testing? Why Test at All?
  • Test Security for Grown-Ups: Enhancing exam integrity through proctoring, recording and monitoring

Bring your colleagues! Group discounts are available. Save up to $200 for additional registrants.

So meet us in enchanting Santa Fe, New Mexico March 21-24 leap into the future of assessment and authoring!

Book your room at the boutique Inn and Spa at Loretto at our special rate, while they are still available. Rates expire February 17, 2017.

Calling all Speakers: Share your story in Santa Fe & become a VIP

Posted by Julie Delazyn

We have officially announced the Questionmark Conference 2017 March 21-14, and we look forward to seeing you at the beautiful Inn and Spa at Loretto for this important learning event.

In order to create a rich and varied conference program, we have opened a call for proposals and invite you to submit your idea for case study presentation or a peer discussion soon.

How do you know if you should submit a proposal? If you can answer yes to any of the questions below, we want to hear from you! Click here to find out more and download the proposal forms!

  • Your experience with Questionmark technologies will help others
  • You have found innovative ways to use online assessments
  • You can explain how you organized your assessment program and what you learned
  • You have gained a lot from previous conferences and want to contribute in 2015
  • You are using assessments to support organizational goals
  • You have a unique application of online or mobile assessments
  • You have integrated Questionmark with another system

We are accepting proposals from now until December 1, so consider what you’d like to contribute.

The Inn and Spa at Loretto

Aside from helping your fellow Questionmark users by sharing your story, please note that presenters will also get some perks:

Presenters and discussion leaders will receive some red carpet treatment — including a special dinner in their honor on Wednesday, March 21. And we award one 50% registration for each case study presentation.

Click here to start the process of becoming a speaker!

Even if you are not sure you’ll attend the conference, we would like to hear from you! And whether you plan to present or not, plan now to have the conference in your budget for 2017.

Learning & Bowling — Looking back on Miami

panel Julie ProfilePosted by Julie Delazyn

We loved seeing so many customers at our conference in Miami earlier this month! We learned so much and greatly appreciate the feedback people shared with us there.

From our exciting panel discussion — which brought together experts from Pacific Gas and Electric, Amazon, The Hartford and HRSG to discuss their valid and reliable processes for assessment — to our networking dinners around Miami Beach, it was a 3-day event packed with excitement and learning opportunities.

You can re-live those moments by browsing through the photos taken during the conference and our Next-Gen release party.smiles

Reporting back from sessions he attended and conversations that took place at Questionmark Conference 2016, Questionmark Chairman John Kleeman has come up with 5 tips to making your assessments valid.Bowling

  • What did you enjoy most about Questionmark Conference 2016?
  • If you didn’t have the chance to join us this year, what would you like to see at next year’s conference?
  • Leave us a comment below and stay in touch!

Item analysis: Selecting items for the test form – Part 2

Austin Fossey-42Posted by Austin Fossey

In my last post, I talked about how item discrimination is the primary statistic used for item selection in classical test theory (CTT). In this post, I will share an example from my item analysis webinar.

The assessment below is fake, so there’s no need to write in comments telling me that the questions could be written differently or that the test is too short or that there is not good domain representation or that I should be banished to an island.

Interested in learning more about item analysis? I will be presenting a series of workshops on this topic at the Questionmark Conference 2016: Shaping the Future of Assessment in Miami, April 12-15.

In this example, we have field tested 16 items and collected item statistics from a representative sample of 1,000 participants. In this hypothetical scenario, we have been asked to create an assessment that has 11 items instead of 16. We will begin by looking at the item discrimination statistics.

Since this test has fewer than 25 items, we will look at the item-rest correlation discrimination. The screenshot below shows the first five items from the summary table in Questionmark’s Item Analysis Report (I have omitted some columns to help display the table within the blog).

IT 2

The test’s reliability (as measured by Cronbach’s Alpha) for all 16 items is 0.58. Note that one would typically need at least a reliability value of 0.70 for low-stakes assessments and a value of 0.90 or higher for high-stakes assessments. When reliability is too low, adding extra items can often help improve the reliability, but removing items with poor discrimination can also improve reliability.

If we remove the five items with the lowest item-rest correlation discrimination (items 9, 16, 2, 3, and 13 shown above), the remaining 11 items have an alpha value of 0.67. That is still not high enough for even low-stakes testing, but it illustrates how items with poor discrimination can lower the reliability of an assessment. Low reliability also increases the standard error of measurement, so by increasing the reliability of the assessment, we might also increase the accuracy of the scores.

Notice that these five items have poor item-rest correlation statistics, yet four of those items have reasonable item difficulty indices (items 16, 2, 3, and 13). If we had made selection decisions based on item difficulty, we might have chosen to retain these items, though closer inspection would uncover some content issues, as I demonstrated during the item analysis webinar.

For example, consider item 3, which has a difficulty value of 0.418 and an item-rest correlation discrimination value of -0.02. The screenshot below shows the option analysis table from the item detail page of the report.

IT2

The option analysis table shows that, when asked about the easternmost state in the Unites States, many participants are selecting the key, “Maine,” but 43.3% of our top-performing participants (defined by the upper 27% of scores) selected “Alaska.” This indicates that some of the top-performing participants might be familiar with Pochnoi Point—an Alaskan island which happens to sit on the other side of the 180th meridian. Sure, that is a technicality, but across the entire sample, 27.8% of the participants chose this option. This item clearly needs to be sent back for revision and clarification before we use it for scored delivery. If we had only looked at the item difficulty statistics, we might never had reviewed this item.

Interested in learning more about item analysis? I will be presenting a series of workshops on this topic at the Questionmark Conference 2016: Shaping the Future of Assessment in Miami, April 12-15. I look forward to seeing you there! Click here to register and learn more about this important learning event.

Item analysis: Selecting items for the test form – Part 1

Austin Fossey-42Regular readers of our blog know that we ran an initial series on item analysis way back in the day, and then I did a second item analysis series building on that a couple of years ago, and then I discussed item analysis in our item development series, and then we had an amazing webinar about item analysis, and then I named my goldfish Item Analysis and wrote my senator requesting that our state bird be changed to an item analysis. So today, I would like to talk about . . . item analysis.

But don’t worry, this is actually a new topic for the blog.

Interested in learning more about item analysis? I will be presenting a series of workshops on this topic at the 2016 Questionmark Conference: Shaping the Future of Assessment in Miami, April 12-15. 

Today, I am writing about the use of item statistics for item selection. I was surprised to find, from feedback we got from many of our webinar participants, that a lot of people do not look at their item statistics until after the test has been delivered. This is a great practice (so keep it up), but if you can try out the questions as unscored field test items before making your final test form, you can use the item analysis statistics to build a better instrument.

When building a test form, item statistics can help us in two ways.

  • They can help us identify items that are poorly written, miskeyed, or irrelevant to the construct.
  • They can help us select the items that will yield the most reliable instrument, and thus a more accurate score.

In the early half of the 20th century, it was common belief that good test instruments should have a mix of easy, medium, and hard items, but this thinking began to change after two studies in 1952 by Fred Lord and by Lee Cronbach and Willard Warrington. These researchers (and others since) demonstrated that items with higher discrimination values create instruments whose total scores discriminate better among participants across all ability levels.

Sometimes easy and hard items are useful for measurement, such as in an adaptive aptitude test where we need to measure all abilities with similar precision. But in criterion-referenced assessments, we are often interested in correctly classifying those participants who should pass and those who should fail. If this is our goal, then the best test form will be one with a range of medium-difficulty items that also have high discrimination values.

Discrimination may be the primary statistic used for selecting items, but item reliability is also occasionally useful, as I explained in an earlier post. Item reliability can be used as a tie breaker when we need to choose between two items with the same discrimination, or it can be used to predict the reliability or score variance for a set of items that the test developer wants to use for a test form.

Difficulty is still useful for flagging items, though an item flagged for being too easy or too hard will often have a low discrimination value too. If an easy or hard item has good discrimination, it may be worth reviewing for item flaws or other factors that may have impacted the statistics (e.g., was it given at the end of a timed test that did not give participants enough time to respond carefully).

In my next post, I will share an example from the webinar of how item selection using item discrimination improves the test form reliability, even though the test is shorter. I will also share an example of a flawed item that exhibits poor item statistics.

Interested in learning more about item analysis? I will be presenting a series of workshops on this topic at the 2016 Questionmark Conference: Shaping the Future of Assessment in Miami, April 12-15. I look forward to seeing you there! Click here to register and learn more about this important learning event.

 

 

Measuring the Impact of Social Learning

julie-smallPosted by Julie Delazyn

“The Social Network” – a movie about Facebook founder Mark Zuckerberg — was a big winner at the recent Golden Globe Awards in Los Angeles. And we’re pleased to say that social networks will be high on the agenda as Questionmark users gather in LA (with black ties nowhere in sight!) March 15 – 18 for the Questionmark 2011 Users Conference.

We’ll be paying considerable attention there to the growing importance of social networks as one aspect of a major shift in how people learn. Many organizations, recognizing that the vast majority of  what we learn comes from firsthand experience and from other people rather than from formal learning and study, are turning to informal and ‘social’ learning initiatives that use portals, blogs, wikis, forums and other tools. At this year’s conference we’ll be highlighting many new features and applications of Questionmark’s Open Assessment Platform that work in concert with many commonly used informal/social learning technologies – from embedded and observational assessment to new user interfaces and tools for content evaluation. (See a sneak peek below — and come to the conference for more on this!)

We’re looking forward to seeing you at the conference in March – we’re making sure the conference agenda will be packed with sessions and events that we’re sure you’ll be able to “like.”

Only two more days to register before the early-bird deadline!