Top 5 Questionmark Videos in 2013

Headshot JuliePosted by Julie Delazyn

As we near the end of the year, we’d like to highlight some of the most popular videos we’ve featured here on the blog in 2013.

We have been posting and sharing many videos from the Questionmark Learning Café. There you can find more than three dozen videos, demos and other resources on everything from quick tutorials to complete webinars about best practices in the use of online surveys, quizzes, tests and exams.

The five most popular videos in 2013… Drumroll, please…

5. Actionable Data And The A-Model
4. Introduction to Questionmark
3. Copy & paste & enhanced question selection in Questionmark Live
2. Assessments Through the Learning Process
1. How to author a Hot Spot Question using Questionmark Live

Thank you for watching, and look for more videos in 2014!

Item Analysis Report – High-Low Discrimination

Austin Fossey-42Posted by Austin Fossey

In our discussion about correlational item discrimination, I mentioned that there are several other ways to quantify discrimination. One of the simplest ways to calculate discrimination is the High-Low Discrimination index, which is included on the item detail views in Questionmark’s Item Analysis Report.

To calculate the High-Low Discrimination value, we simply subtract the percentage of low-scoring participants who got the item correct from the percentage of high-scoring participants who got the item correct. If 30% of our low-scoring participants answered correctly, and 80% of our high-scoring participants answered correctly, then the High-Low Discrimination is 0.80 – 0.30 = 0.50.

But what is the cut point between high and low scorers? In his article, “Selection of Upper and Lower Groups for the Validation of Test Items,” Kelley demonstrated that the High-Low Discrimination index may be more stable when we define the upper and lower groups as participants with the top 27% and bottom 27% of total scores, respectively. This is the same method that is used to define the upper and lower groups in Questionmark’s Item Analysis Report.

The interpretation of High-Low Discrimination is similar to the interpretation of correlational indices: positive values indicate good discrimination, values near zero indicate that there is little discrimination, and negative discrimination indicates that the item is easier for low-scoring participants.

In Measuring Educational Achievement, Ebel recommended the following cut points for interpreting High-Low Discrimination (D):

Capture Blog 18

In Introduction to Classical and Modern Test Theory, Crocker and Algina note that there are some drawbacks to the High-Low Discrimination index. First, it is more common to see items with the same p value having large discrepancies in their High-Low Discrimination values. Second, unlike correlation discrimination indices, High-Low Discrimination can only be calculated for dichotomous items. Finally, the High-Low Discrimination does not have a defined sampling distribution, which means that confidence intervals cannot be calculated, and practitioners cannot determine whether there are statistical differences in High-Low Discrimination values.

Nevertheless, High-Low Discrimination is easy to calculate and interpret, so it is still a very useful tool for item analysis, especially in small-scale assessment. The figure below shows an example of the High-Low Discrimination value on the item detail view of the Item Analysis Report.


High-Low Discrimination value on the item detail page of Questionmark’s Item Analysis Report.

4 Tips for protecting the security of intellectual property

Headshot JulieThe integrity of your tests and test questions is integral to upholding your reputation and standards, as Questionmark Chairman John Kleeman points out in his post: It takes 20 years to build a reputation and five minutes to ruin it.

I have put together four tips to help ensure the security of your intellectual property. To find out more about deploying assessments safely, securely and successfully you can download this complimentary white paper: Delivering Assessments Safely and Securely.

1) Create and administer multiple test forms: Rather than having only one form of the assessment being administered, deliver multiple forms of the same exam to help limit item exposure. If one exam form is breached, the other exam forms can stay in circulation.

2) Restrict and control administration of beta test items: Beta testing questions is an important part of high-stakes assessment, ensuring the psychometric quality of questions before they appear on actual assessments. However, it is important to have a well conceptualized beta test model that limits the exposure of newly developed questions to participants. Beta test questions must be administered in secure environments, in similar conditions to the actual exam. This prevents the exposure of new questions before they appear on an actual assessment. Some rejected beta test questions could be considered for use in exam prep materials.

3) Update exam forms periodically: Letting exam forms become stale can over-expose questions to participants, increasing the likelihood of IP theft. Periodically updating exam forms (e.g., annually) can help limit the exposure of questions. Consider retiring old exam forms and turning them into exam prep materials that can be sold to participants.

4) Produce exam prep materials: Making exam prep materials available to participants before an assessment helps dissuade participants from trying to obtain exam questions via illegal means as they will have access to the type of questions that will be asked on the actual assessment.

Integrating and Connectors – SuccessFactors

Doug Peterson HeadshotPosted By Doug Peterson

In this installment of Integrating and Connectors, I’d like to take a look at SuccessFactors. SuccessFactors is a very popular cloud-based human capital management application suite. It includes modules for succession planning, goals, performance management, recruiting, and – you guessed it – a learning management system (LMS) called Learning (appropriate, no?).

Questionmark assessments can be integrated into SuccessFactors Learning items by publishing them as AICC or SCORM content packages and importing the content package as a content object, which is then included in the learning item. The student logs into SuccessFactors Learning, enrolls in a course, takes in the content, clicks on a link and – voila! – the assessment launches. It’s a seamless experience for the student.

However, our integration with SuccessFactors Learning goes a step further. Learning and Questionmark Enterprise Manager can be connected by a Single Sign-On bridge that allows a Learning administrator to access Questionmark reports directly – no signing into Questionmark EM separately with some other ID and password.

This short video tells the story. Check it out and feel free to contact any of the Questionmark team if you have any questions.

Success Factors

Item Analysis Report – Item-Total Correlation Discrimination

Austin Fossey-42Posted by Austin Fossey

We previously discussed item difficulty (p value), and we talked about how item difficulty relates to item discrimination. Today, let’s move on to Item discrimination—the other item-level statistic commonly used to describe item performance.

While there are several ways to quantify discrimination, test developers commonly use correlational indices to create item discrimination statistics.

In Introduction to Classical and Modern Test Theory, Crocker and Algina explain that item discrimination (sometimes referred to as “differentiation”) is an important tool in the need to identify construct-level differences between individual participants. Those differences are typically measured by a participant’s total score on the assessment.

If we want to quantify the differences between participants with total scores, we need to build an assessment with items that discriminate well between high-scoring and low-scoring participants. We expect our high-scoring participants to have a higher probability of answering any given item correctly. When we observe the opposite (low-scoring participants having a higher probability of answering an item correctly), then the item may not be measuring the same construct that the assessment is purportedly measuring.

The Questionmark Item Analysis Report returns the item-total correlation, which is the Pearson product moment coefficient between the responses for the reported item and the participants’ total test scores. This is a correlational index, so values can range from -1.00 to 1.00. Higher positive values for the item-total correlation indicate that the item is discriminating well between high- and low-performing participants. Negative values mean the opposite: low-performing participants are more likely to get the item correct. Values near zero indicate that the item is not discriminating between high- and low- performing participants—that all participants have similar probabilities of answering the item correctly, regardless of their total assessment score.

The Pearson product moment coefficient is the correlation between an item’s scores and the total test scores, but the total test score is derived partly from that item’s score. By answering the item correctly, a participant automatically increases his or her total score.

This rarely presents an issue when all items are scored the same way and the test form is suitably long (Crocker and Algina suggest 25 items or more). When the form is short, each item proportionally contributes more to the total score, thus creating a self-fulfilling prophecy of sorts: if you get one item correct on a two-item form, you are already well on your way to being a high-scoring participant.

The same is true for assessments where one item is weighted more than others. For example, if one item on the test is scored (0, 20) and the others are scored (0, 1), then people who answer the weighted item correctly automatically score 20 points higher in their total score. This presents a departure from the point we discussed in the item difficulty post, where I mentioned that item discrimination is maximized for items of medium difficulty. When an item is weighted, it may have a very high discrimination even when it has high difficulty, as is shown in the figures below.

Austin Blog 17 First Chart (3)

35 dichotomous items scored (0, 1), plotted by item difficulty and item-total correlation, with highest discrimination values occurring near p = 0.500.

  Austin Blog 17 Second Chart

 35 items plotted by item difficulty and item-total correlation, with one difficult item (p = 0.218) and one easy item (p = 0.908) scored (0, 10). Note that the other items’ correlation values have shifted.

Another consideration for discrimination statistics is the sample size. Small sample sizes will lead to unstable correlation coefficients in any scenario, but they are particularly an issue in the case of item-total correlation. Monika Verzaal at the Netherlands Association of Universities of Applied Sciences notes that small sample sizes can lead to increased item variance, which in turn increases the total test score variance, both of which will affect the magnitude of the item-total correlation.

To accommodate these issues with the Pearson product moment coefficient for the item-total correlation discrimination, some test developers prefer to use the item-rest (sometimes referred to as “item-remainder”) coefficient instead. The interpretation of the statistic is the same, but the item-rest coefficient excludes the reported item’s contribution to the total score when calculating the correlation, so the item variance is not biasing the magnitude of the item-rest coefficient.

Securing online assessment content, exam results and personal information

Joan Phaup 2013 (3)Posted by Joan Phaup

How safe are your online assessment content and exam results?
How secure is the personal information you store?
How would a data breach impact your organization’s reputation?

Secure test delivery and painstaking protection of data are crucial for successful online testing and assessment programs.

Find out in this video about the many measures we take to provide a secure assessment platform and keep information safe.

You will find more information in the Questionmark White Paper, Delivering Assessments Safely and Securely