Intro to high-stakes assessment

Lance bio pic  Posted by

Hello, and welcome to my first blog post for Questionmark. I joined Questionmark in May of 2014 but have just recently become Product Owner for Authoring. That means I oversee the tools we build to help people like you write their questions and assessments.

My professional background in assessment is mostly in the realm of high-stakes testing. That means I’ve worked with organizations that license, certify, or otherwise credential individuals. These include medical/nursing boards, driving standards organizations, software/hardware companies, financial services, and all sorts of sectors where determining competency is important.

With that in mind, I thought I’d kick off my blogging career at Questionmark with a series of posts on the topic of high-stakes assessment.

Now that I’ve riveted your attention with that awesome and no-at-all tedious opening you’re naturally chomping at the bit to learn more, right?


Read on!

High-stakes assessment defined

I think of a high-stakes assessment as having the following traits:

It strongly influences or determines an individual’s ability to practice a profession

Organizations that administer high-stakes assessments operate along a continuum of influence. For example, certifications from SAP or other IT organizations are typically viewed as desirable by employers and may be used as a differentiator when hiring or setting compensation, but are not necessarily required for employment. At the other end of the continuum we have organizations that actually determine a person’s ability to practice a profession. An example is that you must be licensed by a state bar association to practice law in a US state. In between these extremes lie many shades of influence. The key concept here is that the influence is real…from affecting hiring/promotion decisions to flat-out determining if a person can be hired or continue to work in their chosen profession.

It awards credentials that belong to the individual

This is all about scope and ownership. These credentialing organizations almost always award a license/certification to the individual. If you get certified by SAP, that certification is yours even if your employer paid for it.

Speaking about scope, that certificate represents skills that are employer-neutral, and in the case of most IT certifications, the skills are generally unbounded by region as well. A certification acquired in the United States means the same thing in Canada, in Russia, in Mongolia, in Indonesia in…you get the point.

Stan Lee

Okay, so these organizations influence who can work in professions and who can’t. Big whoop, right? Right! It really is a big whoop.

As Stan Lee has told us repeatedly, “Excelsior!”

Hmmm. That’s not the quote I wanted.

I meant, as Stan Lee has told us repeatedly, “With great power comes great responsibility!”*

And these orgs do have great power. They also, in many cases, have powerful members. For example, medical boards in the United States certify elite medical professionals. In all cases, these orgs are simultaneously making determinations about the public good and people’s livelihoods. As a result, they tend to take the process very seriously.

Ok… But what does it all mean?

Glad you asked. Stay tuned for my next post to find out.

Till then, Excelsior!

* So, it turns out that some guy named “Voltaire” said this first. But really, who’s had a bigger impact on the world? Voltaire – if that’s even his real
name – or Stan Lee? 🙂

Evidence that assessments improve learning outcomes

John Kleeman HeadshotPosted by John Kleeman

I’ve written about this research before, but it’s a very compelling example and I think it’s useful as evidence that giving low stakes quizzes during a course correlates strongly with improved learning outcomes.

The study was conducted by two economics lecturers, Dr Simon Angus and Judith Watson, and is titled Does regular online testing enhance student learning in the numerical sciences? Robust evidence from a large data set. It was published in the British Journal of Educational Technology Vol 40 No 2, 255-272 in 2009.

Angus and Watson introduced a series of 4 online, formative quizzes into a business mathematics course, and wanted to determine whether students who took the quizzes learned more and did better on the final exam than those who didn’t. The interesting thing about the study is that they used a statistical technique which allowed them to estimate the effect of several different factors, and isolate the effects of taking the quizzes from the previous mathematical experience of the students, their gender and their general level of effort to determine which impacted the final exam score most.

You can see a summary of their findings in the graph below, which shows the estimated coefficients for four of the main factors, all of which had a statistical significance of p < 0.01.

Factors associated with final exam score graph

You can see from this graph that the biggest factor associated with final exam success was how well students had done in the midterm exam, i.e. how well they were doing in the course generally. But students who took the 4 online quizzes learned from them and did significantly better. The impact of taking or not taking the quizzes was broadly the same as the impact of their prior maths education: i.e. quite reasonable and significant.

We know intuitively that formative quizzes help learning, but it’s nice to see a statistical proof that – to quote the authors – “exposure to a regular (low mark) online quiz instrument has a significant and positive effect on student learning as measured by an end of semester examination”.

Another good resource on the benefits of assessments to check out is the white paper, The Learning Benefits of Questions. In it, Dr. Will Thalheimer of Work-Learning Research reveals research that shows that questions can produce significant learning and performance benefits, potentially improving learning by 150% or more. The white paper is complimentary after registration.

John Kleeman will discuss benefits and good practice in assessments at the 2015 Users Conference in Napa Valley, March 10-13. Register before Dec. 17 and save $200.

Get trustable results : Require a topic score as a prerequisite to pass a test

John Kleeman HeadshotPosted by John Kleeman

If you are taking an assessment to prove your competence as a machine operator, and you get all the questions right except the health and safety ones, should you pass the assessment? Probably not. Some topics can be more important than others, and assessment results should reflect that fact.

In most assessments, it’s acceptable to define a pass or cut score, and all that is required to pass the assessment is for the participant to achieve the passing score or higher. The logic for this is that success on one item can make up for failure on another item,  so skills in one area are substitutable for skills in another. However, there are other assessments where some skills or knowledge are critical, and here you might want to require a passing score or even a 100% score in the key or “golden” topics as well as a pass score for the test as a whole.

This is easy to set up in Questionmark when you author your assessments. When you create the assessment outcome that defines passing the test, you define some topic prerequisites.

Here is an illustrative example, showing 4 topics. As well as achieving the pass score on the test, the participant must achieve 60% in three topics: “Closing at end of day”, “Operations” and “Starting up”, and 100% in one topic: “Safety”.


If you need to ensure that participants don’t pass a test unless they have achieved scores in certain topics, topic prerequisites are the way to achieve this.

Ten tips on reducing test anxiety for online test-takers

Picture of lady biting her nailsJohn Kleeman HeadshotPosted by John Kleeman

I’ve been reading about test anxiety (concern or worry by participants sufficiently severe that it impacts test performance). I’d like to share some tips on how online assessment sponsors can reduce its likelihood.

When creating and delivering tests, you seek to accurately measure knowledge, skills or abilities. Often you want to check competence or mastery for workplace or compliance reasons. If some of your participants have severe test anxiety, this doesn’t just disrupt them, it makes your test less accurate in measuring real performance. You might end up failing someone who is competent, just because anxiety affects their test performance.

Many studies (for example here) report that online tests cause less test anxiety than paper ones. Here are some suggestions on reducing test anxiety:

1. Some people have anxiety about a test because they haven’t mastered the subject being tested. Provide a clear description of what each test covers before the time of the test, and provide study resources or instruction to allow people to master the subject.

2. Test anxiety can also feed on unknowns, for instance on unfamiliarity with the test or believing untrue myths. Share information about the test’s purpose and what you do to make it fair. Also share information about the content: how many questions, how the scoring works, how much time is available and so on. Explain what happens if someone fails – for instance is it possible to retake?

3. It’s hugely valuable to provide practice tests that participants can try out before the real test. This will tell them where they are strong and weak and allow them to gain confidence in a less stressful environment prior to the real test. See my article 10 reasons why practice tests help make perfect exams for other reasons why practice tests are useful.

4. Give participants an opportunity to practice using the same type of computer, mouse, keyboard and user interface as will be used for the real test. This familiarizes them with the test environment and reduces potential anxiety, particularly for those who are less computer literate. If you are using Questionmark to deliver the test, make practice sessions available with the same template settings and the same types of questions. (Sometimes this is done with a fun quiz on a different topic, just to get people accustomed to the user interface.)

5. If you provide guidance to test-takers, point to self-help resources for people who have test anxiety. ETS provide a good resource here for instance. Another resource from the University of California is here.

6. Some self-help resources suggest breathing exercises or other exercises people can follow to reduce tension for people who are anxious about tests. Provide an environment where this is practical and train your test administrators and proctors about the prevalence of test anxiety.

7. If you have a way of encouraging test takers to sleep, take exercise and eat healthily, all these things aid a rational approach to taking a test and reducing anxiety.

8. If it works in your programme, consider whether it’s worth having a series of tests rather than a single test, so there is not a single “make or break” moment for participants. A series of tests can have other benefits too. It makes cheating harder, and by spreading out learning and revision, it can make participants retain the learning better.

9. People with disabilities are more likely to suffer test anxiety. Ensure that your program of accommodations takes this into account. See this helpful article on reducing test anxiety for people with disabilities.

10. Above all, create good quality, fair tests. If you follow good practice in authoring your questions and assessments, then there is less to be anxious about, as the test will be a good measure of performance. See Questionmark’s white paper “Five Steps to Better Tests” for some helpful advice in creating tests.

Many Questionmark users provide very effective practice quizzes and tests which help reduce test anxiety, and I hope these tips are helpful, too.

I’d love to hear additional input or suggestions.

Writing Good Surveys, Part 3: More Question Basics

Doug Peterson HeadshotPosted By Doug Peterson

In part 2 of this series, we looked at several tips for writing good survey questions. To recap:

  • Make sure to ask the right question so that the question returns the data you actually want.
  • Make sure the question is one the respondent can actually answer, typically being about something they can observe or their own personal feelings, but
    not the thoughts/feelings/intentions of others.
  • Make sure the question doesn’t lead or pressure the respondent towards a certain response.
  • Stay away from jargon.
  • Provide an adequate rating scale. Yes/No or Dislike/Neutral/Like may not provide enough options for the respondent to reply honestly.

In this installment, I’d like to look at two more tips. The first is called “barreling”, and it basically refers to asking two or more questions at once. An example might be “The room was clean and well-lit.” Clearly the survey is trying to uncover the respondent’s opinion about the atmosphere of the training room, but it’s conceivable that the room could have been messy yet well-lit, or clean but dimly lit. This is really two questions:

  • The room was clean.
  • The room was well-lit.

I always look for the words “and” and “or” when I’m writing or reviewing questions. If I see an “and” or an “or”, I immediately check to see if I need to split the question out into multiple questions.

The second tip is to keep your questions as short, as clear, and as concise as possible. Long and complex questions tend to confuse the respondent; they get lost along the way. If a sentence contains several commas, phrases or clauses inserted with dashes – you know, like this – or relative or dependent clauses, which are typically set off by commas and words like “which”, it may need to be broken out into several sentences, or may contain unneeded information that can be deleted. (Did you see what I did there?)

In the next few entries in this series, we’re going to take a look some other topics involved in putting together good surveys. These will include how to construct a rating scale as well as some thoughts about the flow of the survey itself. In the meantime, here are some resources you might want to review:

Problems with Survey Questions” by Patti J. Phillips. This covers much of what we looked at in this and the previous post, with several good examples.
Performance-Focused Smile Sheets” by Will Thalheimer. This is an excellent commentary on writing level 2 and level 3 surveys.
Correcting Four Types of Error in Survey Design” by Patti P. Phillips. In this blog article, Patti give a quick run-down of coverage error, sampling error, response rate error, and measurement error.
Getting the Truth into Worplace Surveys” by Palmer Morrel-Samuels in the February 2002 Harvard Business Review. You have to register to read the entire article, or you can purchase it for $6.95 (registration is free).

If you are interested in authoring best practices, be sure to register for the 2014 Questionmark Users Conference  in San Antonio, Texas March 4 – 7. See you there!

Can online quizzes before lectures increase reading by literature students?

John Kleeman HeadshotPosted by John Kleeman

It’s often suggested in higher education circles that computer-assisted assessment is more useful in scientific subjects than in the humanities.

I’d like to share a counter view from some research by Dr Judith Seaboyer at the University of Queensland. She presented a paper at the 2013 International Computer Assisted Assessment conference about how computerized quizzes can help greatly in teaching English Literature.

One challenge of Literature courses is ensuring students read required texts in advance of lectures: Sometimes students struggle to make time for necessary reading, but if they fail to do it, they will likely struggle later on in essays and exams.

Dr. Seaboyer’s solution? Require students to take an online quiz before each lecture. Students must complete the quiz before midnight the night before the first lecture on a text. The quiz, which includes 6 questions chosen at random from a pool of about 15, gives a small amount of course credit. The questions are as Google and eBook search-proof as possible: using different words to those in the text, so they require real reading and understanding.

Here, for example, is a question about Ian McEwan’s Atonement:

Where does Robbie notice a human limb, the memory of which will return to haunt him?

(a) in the fork of a tree

(b) in a Joe Lyons tea house

(c) on the beach at Dunkirk

The right answer is (a), but this would not be easy to identify by searching, as the book mentions a human “leg” not a “limb” and the other answers are plausible. Unless you’ve read the book recently, you will struggle to answer.

Students reported that the online quizzes motivated them to complete assigned reading before the lecture as can be seen in the survey result below:

"The online quiz motivates me to complete assigned reading before the lecture" Mean 4.31 on Strongly Agree to Strongly Disagree Likert Scale

Dr. Seaboyer’s  preliminary research suggests that 83% of first year English students read at least 5 out of 6 books in a course where quizzes were used as against around 45% in a control group.

I’ve seen other examples of quizzes encouraging learners to access learning material that they might otherwise put off until later, and I’d encourage others to consider this approach.

To quote Dr. Seaboyer:

“Computer-assisted assessment can result in more reading and persistent, careful, observant, resilient reading that leads to critical engagement.”

She also believes that this could also apply across a range of other disciplines as well as Literature.