Single Sign-On: Who’s Involved?

Bart Hendrickx SmallPosted by Bart Hendrickx

This is the third post in a series on single sign-on (SSO). Go here for posts 1 and 2:

  1. Single sign-on: secure and easy access
  2. Single Sign-On Pros and Cons

In my first post, I offered this definition for SSO:

Single Sign-On (SSO) for software is the ability for one application, the identity provider, to tell another application, the service provider, who you are.

As you can gather from that definition, SSO involves two parties: the identity provider and the service provider. That is not the complete picture. In many cases, there is also the user and the user agent.

In this post, I explain who those parties are.

Service Provider (SP)

The service provider (SP) is the system a user wants to do something with. For example, Questionmark OnDemand: the user can be a participant who wants to take an assessment. Questionmark OnDemand provides the service of delivering an assessment that the user can take.

When single sign-on is set up, the SP relies on another system to authenticate a user. Therefore, the service provider is also called the relying party (RP). To keep things simple, I will stick to using “SP”.

Identity Provider (IdP)

You know that the SP relies on another system for authentication. That other system is called the identity provider (IdP). The IdP provides identity information on a user to the SP, so that the SP can make a decision to let the user in.

There are many systems that can act as identity providers and some systems specialize in identity. You will recognize many of these: Microsoft Active Directory, Google Account, Facebook, Twitter, LinkedIn—the list is very long.


There is nothing special to user: this is simply the person who wants to do something, such as the participant who wants to take an assessment. Well, the user does not need to be a person; it can also be a system. I will continue using the concept of “person” when I talk about “user”, for simplicity. (Did I already tell you I like to keep things simple?)

This party is sometimes called principal.

User Agent

The user agent is a piece of software, an application, which acts on behalf of the user. That sounds complicated, so let me give you an example. Your web browser is a user agent. When you visit a web page, your browser requests content on your behalf. Is an image referenced in the content? Your browser downloads it, on your behalf. Do you want to submit a form, such as an order with payment details? Your browser sends the information to the website, on your behalf.

In the context of SSO, the user agent often is a browser or an application that can access something on the Internet, such as an app on your smartphone to read emails or access a social network.

The Parties Working Together

Let me bring this together in a picture.


You see a group of users, say, participants who want to take an assessment. They use their web browser, which is a user agent, to connect to the service provider; Questionmark OnDemand in this example.

The service provider relies on an identity provider to say who these users are. In this example, instead of talking directly to the identity provider, the service provider talks to the user agent (browser), who talks to the identity provider. “Who is this user?” the user agent asks the identity provider. The user may need to enter a username and password, on a page which is, you guessed it, displayed by the user agent.

When the identity provider successfully authenticates the user, it responds to the user agent: “This is Jane Doe.” The user agent passes that information on to the service provider, who then decides to give access: “This is Jane Doe and she is a participant; I will make this assessment available to her.”

How can the service provider and identity provider talk to each other, in this case indirectly through the user agent (browser)? They speak a shared language; a protocol that is used to exchange identity information. There are several popular such protocols. I will discuss them in a following post.

Develop Better Tests with Item Analysis [New eBook]

Posted by Chloe Mendonca

Item Analysis is probably the most important tool for increasing test effectiveness.  In order to write items that accurately and reliably measure what they’re intended to, you need to examine participant responses to each item. You can use this information to improve test items and identify unfair or biased items.

So what’s the process for conducting an item analysis? What should you be looking for? How do you determine if a question is “good enough”?

Questionmark has just published a new eBook “Item Analysis Analytics, which answers these questions. The eBook shares many examples of varying statistics that you may come across item analysis ebookin your own analyses.

Download this eBook to learn about these aspects of analytics:

  • the basics of classical test theory and item analysis
  • the process of conducting an item analysis
  • essential things to look for in a typical item analysis report
  • whether a question “makes the grade” in terms of psychometric quality

This eBook is available as a PDF and ePUB suitable for viewing on a variety of mobile devices and eReaders.

I hope you enjoy reading it!

7 Strategies to Shrink Satisficing & Improve Survey Results

John Kleeman Headshot

Posted by John Kleeman

My previous post Satisficing: Why it might as well be a four-letter word explained that satisficing on a survey is when someone answers survey questions adequately but not as well as they can. Typically they just fill in questions without thinking too hard. As a commenter on the blog said: “Interesting! I have been guilty of this, didn’t even know it had a name!”

Examples of satisficing behavior are skipping questions or picking the first answer that makes some kind of sense. Satisficing is very common.  As explained in the previous blog, some reasons for it are participants not being motivated to answer well, not having the ability to answer well, them finding the survey too hard or them simply becoming fatigued at too long a survey.

Satisficing is a significant cause of survey error, so here are 7 strategies for a survey author to reduce satisficing:

1. Keep surveys short. Even the keenest survey respondent will get tired in a long survey and most of your respondents will probably not be keen. To get better results, make the survey as short as you possibly can.Bubble-Sheet---Printing-and-Scanning_2

2. Keep questions short and simple. A long and complex question is much more likely to get a poor quality answer.  You should deconstruct complex questions into shorter ones. Also don’t ask about events that are difficult to remember. People’s memory of the past and of the time things happened is surprisingly fragile, and if you ask someone about events weeks or months ago, many will not recall well.

3. Avoid agree/disagree questions. Satisficing participants will most likely just agree with whatever statement you present. For more on the weaknesses of these kind of questions, see my blog on the SAP community network: Strongly Disagree? Should you use Agree/Disagree in survey questions?

4. Similarly remove don’t know options. If someone is trying to answer as quickly as possible, answering that they don’t know is easy for them to do, and avoids thinking about the questions.

5. Communicate the benefit of the survey to make participants want to answer well. You are doing the survey for a good reason.  Make participants believe the survey will have positive benefits for them or their organization. Also make sure each question’s results are actionable. If the participant doesn’t feel that spending the time to give you a good answer is going to help you take some useful action, why should they bother?

6. Find ways to encourage participants to think as they answer. For example, include a request to ask participants to carefully deliberate – it could remind them to pay attention. It can also be helpful to occasionally ask participants to justify their answers – perhaps adding a text comment box after the question explaining why they answered that way. Adding comment boxes is very easy to do in Questionmark software.

7. Put the most important questions early on. Some people will satisfice and they are more likely to do it later on in the survey. If you put the questions that matter most early on, you are more likely to get good results from them.

There is a lot you can do to reduce satisficing and encourage people to give their best answers. I hope these strategies help you shrink the amount of satisficing your survey participants do, and in turn give you more accurate results.

Item Development Tips For Defensible Assessments

Julie ProfilePosted by Julie Delazyn

Whether you work with low-stakes assessments, small-scale classroom assessments or large-scale, high-stakes assessment, understanding and applying some basic principles of item development will greatly enhance the quality of your results.

What began as a popular 11-part blog series has morphed into a white paper: Managing Item Development for Large-Scale Assessment, which offers sound advice on how-to organize and execute item development steps that will help you create defensible assessments. These steps include:   Item Dev.You can download your copy of the complimentary white paper here: Managing Item Development for Large-Scale Assessment

Online Proctoring: FAQs

John Kleeman HeadshotPosted by John Kleeman

Online proctoring was a hot-button topic at Questionmark’s annual Users Conference. And though we’ve discussed the pros and cons in this blog and even offered an infographic highlighting online versus test-center proctoring, many interesting questions arose during the Ensuring Exam Integrity with Online Proctoring  session I presented with Steve Lay at Questionmark Conference 2016.

I’ve compiled a few of those questions and offered answers to them. For context and additional information, make sure to check out a shortened version of our presentation. If you have any questions you’d like to add to the list, comment below!

What control does the online proctor have on the exam?

With Questionmark solutions, the online proctor can:

  • Converse with the participant
  • Pause and resume the exam
  • Give extra time if needed
  • Terminate the exam

What does an online proctor do if he/she suspects cheating?

Usually the proctor will terminate the exam and file a report to the exam sponsor.

What happens if the exam is interrupted, e.g. by someone coming in to the room?

This depends on your security protocols. Some organizations may decide  to terminate the exam and require another attempt. In some cases, if it seems an honest mistake, the organization may decide that the proctor can use discretion to permit the exam to continue.

Which is more secure, online or face-to-face proctoring?online proctoring

On balance, they are about equally secure.

Unfortunately there has been a lot of corruption with face-to-face proctoring, and online proctoring makes it much harder for participant and proctor to collude as there is no direct contact, and all communication can be logged.

But if the proctors are honest, it is easier to detect cheating aids in a face-to-face environment than via a video link.

What kind of exams is online proctoring good for?

Online proctoring works well for exams where:

  • The stakes are high and so you need the security of a proctor
  • Participants are in many different places, making travel to test centers costly
  • Participants are computer literate – have and know how to use their own PCs
  • Exams take 2-3 hours or less

If your technology or subject area changes frequently, then online proctoring is particularly good because you can easily give more frequent exams, without requiring candidates to travel.

What kind of exams is online proctoring less good for?

Online proctoring is less appropriate for exams where:

  • Exams are long and participants needs breaks
  • Exams where participants are local and it’s easy to get them into one place to take the exam
  • Participants do not have access to their own PC and/or are not computer literate

How do you prepare for online proctoring?

Here are some preparation tasks:

  • Brief and communicate with your participants about online proctoring
  • Define clearly the computer requirements for participants
  • Agree what happens in the event of incidents – e.g. suspected cheating, exam interruptions
  • Agree what ID is acceptable for participants and whether ID information is going to be stored
  • Make a candidate agreement or honor code which sets out what you expect from people to encourage them to take the exam fairly

I hope these Q&A and the linked presentation are interesting. You can find out more about Questionmark’s online proctoring solution here.

The tips and tools you need to get the most out of your assessments [Webinars]

Chloe Mendonca

Posted by Chloe Mendonca

What’s the big deal about assessments anyway? Though they’ve been around for decades, the assessment and eLearning industry is showing no sign of slowing down. Organisations large and small are using a wide variety of assessment types to measure knowledge, skills, abilities, personality and more.

Join us for one of our upcoming 60-minute webinars and discover the tools, technologies and processes organisations are using worldwide to increase the effectiveness of their assessment programs.

How to transform recruitment and hiring with online testing

This webinar, presented by Dr. Glen Budgell, Senior Strategic HR Advisor at Human Resource Systems Group (HRSG), will discuss the importance and effectiveness of using online testing within HR. This is a must-attend event for anyone exploring the potential of online testing for improving recruitment.

How to Build a Highly Compliant Team in a Fast Moving Market

Organisations across highly regulated industries contend with both stringent regulatory requirements and the need for rigorous asessment programs.  With life, limb, and livelihood on the line, safety and compliance requires much more than “checking a box”. During this webinar, hosted by Questionmark and SAP we will examine ways in which organisations can use online assessment to enhance and strengthen their compliance initiatives.

Introduction to Questionmark’s Assessment Management System

Join us for a live demonstration and learn how Questionmark’s online assessment platform provides organisations with the tools to efficiently develop and deliver assessments.

You can also catch this introductory webinar in Portuguese!

Conhecendo a Questionmark e seu Portal de Gestão de Avaliações [Portuguese]