Beta Testing Questions: Methods and Best Practices

greg_pope-150x1502

Posted by Greg Pope

I had the good fortune of presenting a few sessions at the Questionmark 2010 Users Conference in sunny Miami a couple of weeks ago. It was a great opportunity to catch up with customers and learn about the priorities organizations are focusing on.

In one of my best practice sessions there was a great deal of interest in the topic of beta testing, so I thought I would put together a blog article on this in case others were interested.

Beta testing can be defined as gathering psychometric information regarding newly created questions in order to inform the creation of actual exams. Newly developed questions that have gone through the necessary editing and review processes are administered to representative samples of participants, either in advance of or during an actual high-stakes assessment. Psychometric information regarding the new questions is collected and used to build the actual assessments. Questions that have been beta tested are screened to ensure that they meet certain quality benchmarks (e.g., all questions fall into a certain range of difficulty, all questions have acceptable discrimination). These beta tested questions are then used to create the assessments built to specific structure criteria (e.g., there is an appropriate spread of question difficulty, a targeted mean test score is created, more questions are included on the assessment near the pass score if the assessment is criterion referenced, etc.).

A summary graphic describing the general beta testing process is included below:

raphic describing the general beta testing process

There are a number of common models for beta testing questions, two of the most common are:


models for beta testing questions

Want more details? Questionmark software support plan customers can learn more about beta testing from our best practice guide on this topic. See our Best Practice Guide Index.

2 Responses to “Beta Testing Questions: Methods and Best Practices”

  1. […] are beta testing questions try to obtain samples that reflect that population of test takers. In a previous blog post I talked more about beta […]

  2. Joan Bartel says:

    Hello Greg:

    I am a business development director for the Richmond Indiana Certified Technology Park. I have a client who has an “advanced manufactured” product and will be applying for funding. Do you ever assist a company in writing a beta test questionnaire. We do not have a CTP Business Director to assist. The client would have to write the questionnaire and they are not skilled to do this either.

    Suggestions…thoughts? Thank you and have a wonderful day.

Leave a Reply