Secrets to Measuring & Enhancing Learning Results: Webinar

Julie ProfilePosted by Julie Delazyn

Research has shown that assessments play an important role on learning and retention — and the benefits vary before, during and after a learning experience. No matter where learning occurs, the goal remains the same: ensuring people have the knowledge, skills and abilities to perform well.

So, how can you use assessments to measure and enhance learning within your organization?

Check out our newest 30-minute webinar – and register today!

  • The Secrets to Measuring and Enhancing Learning Results
  • Date & Time: Wed, Dec 7  at 4:00 p.m. UK GMT / 11:00 a.m. US EDT

Join us as we discuss the important role assessments play within the learning process and explore the benefits of using them before, during and after learning. We’ll also give you some useful pointers and resources to take away.

Register for the webinar now. We look forward to seeing you at the session!

Online Proctoring: FAQs

John Kleeman HeadshotPosted by John Kleeman

Online proctoring was a hot-button topic at Questionmark’s annual Users Conference. And though we’ve discussed the pros and cons in this blog and even offered an infographic highlighting online versus test-center proctoring, many interesting questions arose during the Ensuring Exam Integrity with Online Proctoring  session I presented with Steve Lay at Questionmark Conference 2016.

I’ve compiled a few of those questions and offered answers to them. For context and additional information, make sure to check out a shortened version of our presentation. If you have any questions you’d like to add to the list, comment below!

What control does the online proctor have on the exam?

With Questionmark solutions, the online proctor can:

  • Converse with the participant
  • Pause and resume the exam
  • Give extra time if needed
  • Terminate the exam

What does an online proctor do if he/she suspects cheating?

Usually the proctor will terminate the exam and file a report to the exam sponsor.

What happens if the exam is interrupted, e.g. by someone coming in to the room?

This depends on your security protocols. Some organizations may decide  to terminate the exam and require another attempt. In some cases, if it seems an honest mistake, the organization may decide that the proctor can use discretion to permit the exam to continue.

Which is more secure, online or face-to-face proctoring?online proctoring

On balance, they are about equally secure.

Unfortunately there has been a lot of corruption with face-to-face proctoring, and online proctoring makes it much harder for participant and proctor to collude as there is no direct contact, and all communication can be logged.

But if the proctors are honest, it is easier to detect cheating aids in a face-to-face environment than via a video link.

What kind of exams is online proctoring good for?

Online proctoring works well for exams where:

  • The stakes are high and so you need the security of a proctor
  • Participants are in many different places, making travel to test centers costly
  • Participants are computer literate – have and know how to use their own PCs
  • Exams take 2-3 hours or less

If your technology or subject area changes frequently, then online proctoring is particularly good because you can easily give more frequent exams, without requiring candidates to travel.

What kind of exams is online proctoring less good for?

Online proctoring is less appropriate for exams where:

  • Exams are long and participants needs breaks
  • Exams where participants are local and it’s easy to get them into one place to take the exam
  • Participants do not have access to their own PC and/or are not computer literate

How do you prepare for online proctoring?

Here are some preparation tasks:

  • Brief and communicate with your participants about online proctoring
  • Define clearly the computer requirements for participants
  • Agree what happens in the event of incidents – e.g. suspected cheating, exam interruptions
  • Agree what ID is acceptable for participants and whether ID information is going to be stored
  • Make a candidate agreement or honor code which sets out what you expect from people to encourage them to take the exam fairly

I hope these Q&A and the linked presentation are interesting. You can find out more about Questionmark’s online proctoring solution here.

Will testing employees reduce fines for compliance errors?

John Kleeman Headshot

Posted by John Kleeman

If a bank faces a fine of millions for money laundering and then can prove, defensibly, that the ‘accused’ had passed competency tests, would that reduce or eliminate the fine? More generally, suppose employees do something wrong and the corporation is facing a regulatory fine. Does it make a difference if those employees were certified? Is it a defence against regulatory action that you took all the measures you could to prevent error?

We are asked this question from time to time, and the answer varies considerably by regulator and by offence. But in general having competent/certified people and good compliant processes will reduce the impact to the corporation of making a compliance mistake. In some cases it might eliminate a fine, but usually not.

Here are three specific examples where a good compliance program can reduce or eliminate fines.

Prosecutors should therefore attempt to determine whether a corporation’s compliance program is merely a “paper program” or whether it was designed, implemented, reviewed … in an effective manner. In addition, prosecutors should determine whether the corporation has provided for a staff sufficient to audit, document, analyze, and utilize the results of the corporation’s compliance efforts. Prosecutors also should determine whether the corporation’s employees are adequately informed about the compliance program and are convinced of the corporation’s commitment to it. This will enable the prosecutor to make an informed decision as to whether the corporation has adopted and implemented a truly effective compliance program that … may result in a decision to charge only the corporation’s employees and agents or to mitigate charges or sanctions against the corporation.

  • The UK Ministry of Justice guidance on the Bribery Act recommends communication and training around bribery and says that “it is a full defence for an organisation to prove that despite a particular case of bribery it nevertheless had adequate procedures in place to prevent persons associated with it from bribing.”
  • Similarly, in Spain, the Spanish criminal code has been updated so that companies may avoid criminal prosecution if they have an effective compliance program in effect including evidence that employees have had sufficient training in the compliance program.

Fines rising to over one billion pounds in 2014 and nearly one billion pounds in 2015In general, the issue is more diffuse. For example, the UK Financial Conduct Authority, which has issued many huge fines over the years (see graph right), does not seem to explicitly reduce fines based on compliance measures.

But its Penalties Manual does say that fines should be increased if the actions are deliberate or reckless or if the breach resulted from systematic weaknesses in the firm’s procedures. Equally, if the breach was inadvertent and there is no evidence that the breach indicates a widespread problem or weakness, the fine might be lower.

So how best to summarize this?

The biggest benefit of a programme for competency testing for employees is that, in conjunction with other compliance measures, it will reduce the chances of an infraction in the first place.

Having certified or competent people is not a “get out of jail free” card but if part of a professional compliance programme, it will help with many regulators in mitigating financial penalties after an infraction.

Agree or disagree? 10 tips for better surveys — Part 2

John Kleeman HeadshotPosted by John Kleeman

In my first post in this series, I explained that survey respondents go through a four-step process when they answer each question: comprehend the question, retrieve/recall the information that it requires, make a judgement on the answer and then select the response. There is a risk of error at each step. I also explained the concept of “satisficing”, where participants often give a satisfactory answer rather than an optimal one – another potential source of error.

Today, I’m offering some tips for effective online attitude survey design, based on research evidence. Following these tips should help you reduce error in your attitude surveys.

Tip #1 – Avoid Agree/Disagree questions

Although these are one of the most common types of questions used in surveys, you should try to avoid questions which ask participants whether they agree with a statement.

There is an effect called acquiescence bias, where some participants are more likely to agree than disagree. It seems from the research that some participants are easily influenced and so tend to agree with things easily. This seems to apply particularly to participants who are more junior or less well educated, who may tend to think that what is asked of them might be true. For example Krosnick and Presser state that across 10 studies, 52 percent of people agreed with an assertion compared to 42 percent of those disagreeing with its opposite. If you are interested in finding more about this effect, see this 2010 paper by Saris, Revilla, Krosnick and Schaeffer.

Satisficing – where participants just try to give a good enough answer rather than their best answer – also increases the number of “agree” answers.

For example, do not ask a question like this:

My overall health is excellent. Do you:

  • Strongly Agree
  • Agree
  • Neither Agree or Disagree
  • Disagree
  • Strongly Disagree

Instead re-word it to be construct specific:

How would you rate your health overall?

  • Excellent
  • Very good
  • Good
  • Fair
  • Bad
  • Very bad

 

Tip #2 – Avoid Yes/No and True/False questions

For the same reason, you should avoid Yes/No questions and True/False questions in surveys. People are more likely to answer Yes than No due to acquiescence bias.

Tip #3 – Each question should address one attitude only

Avoid double-barrelled questions that ask about more than one thing. It’s very easy to ask a question like this:

  • How satisfied are you with your pay and work conditions?

However, someone might be satisfied with their pay but dissatisfied with their work conditions, or vice versa. So make it two separate questions.

Tip #4 – Minimize the difficulty of answering each question

If a question is harder to answer, it is more likely that participants will satisfice – give a good enough answer rather than the best answer. To quote Stanford Professor  Jon Krosnick, “Questionnaire designers should work hard to minimize task difficulty”.  For example:

  • Use as few words as possible in question and responses.
  • Use words that all your audience will know.
  • Where possible, ask questions about the recent past not the distant past as the recent past is easier to recall.
  • Decompose complex judgement tasks into simpler ones, with a single dimension to each one.
  • Where possible make judgements absolute rather than relative.
  • Avoid negatives. Just like in tests and exams, using negatives in your questions adds cognitive load and makes the question less likely to get an effective answer.

The less cognitive load involved in questions, the more likely you are to get accurate answers.

Tip #5 – Randomize the responses if order is not importantSetting choices to be shuffled

The order of responses can significantly influence which ones get chosen.

There is a primacy effect in surveys where participants more often choose the first response than a later one. Or if they are satisficing, they can choose the first response that seems good enough rather than the best one.

There can also be a recency effect whereby participants read through a list of choices and choose the last one they have read.

In order to avoid these effects, if your choices do not have a clear progression or some other reason for being in a particular order, randomize them.  This is easy to do in Questionmark software and will remove the effect of response order on your results.

Here is a link to the next segment of this series: Agree or disagree? 10 tips for better surveys — part 3

Is There Value in Reporting Subscores?

Austin Fossey-42Posted by Austin Fossey

The decision to report subscores (reported as Topic Scores in Questionmark’s software) can be a difficult one, and test developers often need to respond to demands from stakeholders who want to bleed as much information out of an instrument as they can. High-stakes test development is lengthy and costly, and the instruments themselves consume and collect a lot of data that can be valuable for instruction or business decisions. It makes sense that stakeholders want to get as much mileage as they can out of the instrument.

It can be anticlimactic when all of the development work results in just one score or a simple pass/fail decision. But that is after all what many instruments are designed to do. Many assessment models assume unidimensionality, so a single score or classification representing the participant’s ability is absolutely appropriate. Nevertheless, organizations often find themselves in the position of trying to wring out more information. What are my participants’ strengths and weaknesses? How effective were my instructors? There are many ways in which people will try to repurpose an assessment.

The question of whether or not to report subscores certainly falls under this category. Test blueprints often organize the instrument around content areas (e.g., Topics), and these lend themselves well to calculating subscores for each of the content areas. From a test user perspective, these scores are easy to interpret, and they are considered valuable because they show content areas where participants perform well or poorly, and because it is believed that this information can help inform instruction.

But how useful are these subscores? In their article, A Simple Equation to Predict a Subscore’s Value, Richard Feinberg and Howard Wainer explain that there are two criteria that must be met to justify reporting a subscore:

  • The subscore must be reliable.
  • The subscore must contain information that is sufficiently different from the information that is contained by the assessment’s total score.

If a subscore (or any score) is not reliable, there is no value in reporting it. The subscore will lack precision, and any decisions made on an unreliable score might not be valid. There is also little value if the subscore does not provide any new information. If the subscores are effectively redundant to the total score, then there is no need to report them. The flip side of the problem is that if subscores do not correlate with the total score, then the assessment may not be unidimensional, and then it may not make sense to report the total score. These are the problems that test developers wrestle with when they lie awake at night.

Excerpt from Questionmark’s Test Analysis Report showing low reliability of three topic scores.

As you might have guessed from the title of their article, Feinberg and Wainer have proposed a simple, empirically-based equation for determining whether or not a subscore should be reported. The equation yields a value that Sandip Sinharay and Shelby Haberman called the Value Added Ratio (VAR). If a subscore on an assessment has a VAR value greater than one, then they suggest that this justifies reporting it. All of the VAR values that are less than one, should not be reported. I encourage interested readers to check out Feinberg and Wainer’s article (which is less than two pages, so you can handle it) for the formula and step-by-step instructions for its application.

 

Intro to high-stakes assessment

Lance bio pic  Posted by

Hello, and welcome to my first blog post for Questionmark. I joined Questionmark in May of 2014 but have just recently become Product Owner for Authoring. That means I oversee the tools we build to help people like you write their questions and assessments.

My professional background in assessment is mostly in the realm of high-stakes testing. That means I’ve worked with organizations that license, certify, or otherwise credential individuals. These include medical/nursing boards, driving standards organizations, software/hardware companies, financial services, and all sorts of sectors where determining competency is important.

With that in mind, I thought I’d kick off my blogging career at Questionmark with a series of posts on the topic of high-stakes assessment.

Now that I’ve riveted your attention with that awesome and no-at-all tedious opening you’re naturally chomping at the bit to learn more, right?

Right?

Read on!

High-stakes assessment defined

I think of a high-stakes assessment as having the following traits:

It strongly influences or determines an individual’s ability to practice a profession

Organizations that administer high-stakes assessments operate along a continuum of influence. For example, certifications from SAP or other IT organizations are typically viewed as desirable by employers and may be used as a differentiator when hiring or setting compensation, but are not necessarily required for employment. At the other end of the continuum we have organizations that actually determine a person’s ability to practice a profession. An example is that you must be licensed by a state bar association to practice law in a US state. In between these extremes lie many shades of influence. The key concept here is that the influence is real…from affecting hiring/promotion decisions to flat-out determining if a person can be hired or continue to work in their chosen profession.

It awards credentials that belong to the individual

This is all about scope and ownership. These credentialing organizations almost always award a license/certification to the individual. If you get certified by SAP, that certification is yours even if your employer paid for it.

Speaking about scope, that certificate represents skills that are employer-neutral, and in the case of most IT certifications, the skills are generally unbounded by region as well. A certification acquired in the United States means the same thing in Canada, in Russia, in Mongolia, in Indonesia in…you get the point.

Stan Lee

Okay, so these organizations influence who can work in professions and who can’t. Big whoop, right? Right! It really is a big whoop.

As Stan Lee has told us repeatedly, “Excelsior!”

Hmmm. That’s not the quote I wanted.

I meant, as Stan Lee has told us repeatedly, “With great power comes great responsibility!”*

And these orgs do have great power. They also, in many cases, have powerful members. For example, medical boards in the United States certify elite medical professionals. In all cases, these orgs are simultaneously making determinations about the public good and people’s livelihoods. As a result, they tend to take the process very seriously.

Ok… But what does it all mean?

Glad you asked. Stay tuned for my next post to find out.

Till then, Excelsior!

* So, it turns out that some guy named “Voltaire” said this first. But really, who’s had a bigger impact on the world? Voltaire – if that’s even his real
name – or Stan Lee? 🙂