International University gains scalability, advanced authoring, single sign-on and more!

ChloePosted by Chloe Mendonca

Earlier this month I had the pleasure of speaking with The International University of Languages and Media (IULM) based out of Milan, Italy. IULM has been using Questionmark’s Assessment Management System for almost 10 years, so I was also keen to see what prompted their recent move from managing their assessment program on-premise to Questionmark OnDemand (Questionmark’s SaaS cloud-based solution).iulm

IULM serves approximately 5,000 students enrolled across a range of postgraduate and undergraduate courses including interpretation, translation and cultural studies; communication, public relations and advertising, and arts and tourism.

Patrizia Lettieri, the digital platform administrator in the university’s IT department explains why after 10 years, the IULM decided to make the move:  “Our teachers liked using Questionmark; we didn’t want to change that, but there was definitely a need for a more scalable and innovative solution that would match the direction the university was headed in terms of growth. Security and reporting were also very important to us and there is nothing on par with Questionmark OnDemand in regards to security and reporting.”

Here are some highlights from our conversation. You can read the entire case study here.

Advanced authoring
A big draw for IULM in using Questionmark was the ease of use when it came to assessment authoring and improving the way items are written and also the efficiency with which they can manage their item bank. Patrizia elaborates: “The variety of item types and ability to embed images, videos and links into our questions and feedback enable us to further develop the way we are helping our students to learn. Meta tagging of items is also very simple. Before using OnDemand we had to separate these into different folders and our teachers used to delete items that were invalid. Now we can easily assign the appropriate meta tags and modify the status of questions to retired if they are invalid.”

Scalability and save-as-you-go
iulm_campusBefore using Questionmark OnDemand IULM had a few challenges when it came to assessing large numbers of students.  With OnDemand, IULM can reliably deliver their assessments to multiple classes of students. And in the event that something does go wrong — such as a power cut or other technical failure — Questionmark’s save-as-you-go is there to ensure student answers are automatically saved at regular intervals. “My priority is the students; they are our customers and I want them to be comfortable,” adds Patrizia. “With OnDemand everything runs smoothly and our students are more comfortable and confident when taking their assessments.”

Single Sign-On
It used to be that every day, staff and students at the university were having to interact with a variety of applications requiring individuals to remember different credentials for the various applications.  “With Questionmark OnDemand, we were able to take advantage of single sign-on (SSO). For a forward-thinking technological university, improving access for both students and staff just made sense for us.” SSO also saves IULM time on the administrative side. Once students have completed their studies and leave the university, staff no longer have to remove their access to each application but only need to remove each student’s identity provider account. Without that account, the student can no longer log on to any of the linked applications.

Multilingual delivery
Being an international university, having students who speak different languages is a common scenario. Questionmark’s multilingual delivery helps IULM accommodate students whose first language is not Italian or English by enabling them to change the delivery interface to their preferred language. Having the navigation controls in a participant’s own language can often make students feel more at ease during the test taking experience.

If you’re interested in trying out Questionmark OnDemand, request a demo today and one of our team will show you how simple assessment management can be.

SAML 101: How it works

Bart Hendrickx SmallPosted by Bart Hendrickx

In my last post, I wrote about what SAML is. In this one, I’ll offer a use case to put it into context. There are a number of scenarios where SAML can be used, but I will stick toSSO3 login (authentication) that is initiated by the service provider. I’ll use Questionmark OnDemand as an example of a SP that can work with SAML. Our fictitious customer has an identity provider that is internally hosted behind a firewall, inaccessible from the outside world. Users at the customer’s company can go on the Internet; therefore, they can also take Questionmark OnDemand assessments.

User Jane Doe wants to connect to Questionmark OnDemand, to take an assessment that was scheduled to her. She browses to her company’s OnDemand area, which had been set up to authenticate via SAML. Through the federation metadata, Questionmark OnDemand knows which identity provider to ask for those authentication details. But it cannot talk to the IdP directly. Instead, it creates a SAML request which the web browser passes on to the IdP. Jane Doe’s computer is on the internal network and can access the IdP. The request is forwarded to the IdP, which accepts it because it knows about the service provider (SP), i.e. the customer’s OnDemand area—also possible thanks to the federation metadata.

Jane Doe is already logged on to the IdP: she opened her company’s intranet page this morning, which required her to authenticate, and that session is still active in her browser. So when the IdP gets a request: “Who is this user?”, it already knows the answer: “This is Jane Doe.” The IdP prepares a SAML response and includes a number of attributes, such as Jane Doe’s email address and hire date. All those data form an assertion, which is part of the response.

Again, Jane Doe’s browser plays a key role. It receives the SAML response with the assertion from the IdP and passes it on to the customer’s OnDemand area, which then reads the response. The OnDemand area confirms that this information comes from its trusted IdP and sees that this is Jane Doe. and that an assessment has been scheduled to her. Jane Doe now has access to the OnDemand area and can take the assessment.

For Jane Doe, this all happens seamlessly. She may see her browser redirect to other URLs a few times, when it is relaying information from the SP to the IdP and vice versa, but the entire process usually only takes a couple of seconds.

In a future post, I will explain what SAML requests and responses do and do not contain. Stay tuned!

SAML 101

Bart Hendrickx SmallPosted by Bart Hendrickx

As I mentioned in my previous post on SSO, Single Sign-On: Who’s Involved?, we’ll take a look at SAML to understand what it is and how it’s used with SSO. In this post I’ll explain what SAML is, and I will offer an example use case in my next post.


So, What Is SAML?

SAML, or Security Assertion Markup Language, is a protocol that allows systems to exchange authentication data on users. (It facilitates other use cases as well, but I will focus on authentication.) What does that mean? It means that one system can ask: “Who is this user?” and another system can answer: “This is Jane Doe.” As I mentioned in my post previous post, I am talking about the service provider (SP) and identity provider (IdP) respectively.

Service providers (SP) in this context can be any software system with which you can do something, such as sending and receiving email, tracking projects or delivering assessments. Similarly, an identity provider (IdP) can be any software system that contain data on users that you can use to determine who those users are.

If you manage an SP, you probably don’t want just any IdP telling you who someone is. You will typically trust only one or a few IdPs. And if you are in charge of an IdP, you will likewise prefer to send user data only to those SPs you know and trust. To accomplish that, the SP and IdP exchange data allowing them to establishing a trust relationship. Those data are often called federation metadata, federation referring to the fact that there is an alliance between the different systems.

SAML is a popular protocol to set up such federations between service providers and identity providers. Look up SAML in your favorite search engine and you will get many results. One of its advantages is that it is extensible, meaning that you can exchange information that is relevant to your situation. For example, do you have an IdP that stores the hire date for an employee (or enrollment date of a student)? Do you want to share those data with an SP so that it can decide whether the user is allowed to access a certain resource? Then you can set up the federation in such a way that the IdP will send an attribute for hire (or enrollment) date to the SP.

Another advantage, and it is a huge one, is that SAML can be used in situations where the IdP and SP cannot talk to each other, for example because they are on different networks. You may have an IdP running on your internal network, behind a firewall. Your SP may be available in the cloud, as is the case with Questionmark OnDemand. The SP cannot talk to the IdP because it cannot “see” it. However, that’s not a problem for SAML. In my next post, we’ll take a look at a typical use case so we can see the practicality of using SAML with SSO.


Online Proctoring: FAQs

John Kleeman HeadshotPosted by John Kleeman

Online proctoring was a hot-button topic at Questionmark’s annual Users Conference. And though we’ve discussed the pros and cons in this blog and even offered an infographic highlighting online versus test-center proctoring, many interesting questions arose during the Ensuring Exam Integrity with Online Proctoring  session I presented with Steve Lay at Questionmark Conference 2016.

I’ve compiled a few of those questions and offered answers to them. For context and additional information, make sure to check out a shortened version of our presentation. If you have any questions you’d like to add to the list, comment below!

What control does the online proctor have on the exam?

With Questionmark solutions, the online proctor can:

  • Converse with the participant
  • Pause and resume the exam
  • Give extra time if needed
  • Terminate the exam

What does an online proctor do if he/she suspects cheating?

Usually the proctor will terminate the exam and file a report to the exam sponsor.

What happens if the exam is interrupted, e.g. by someone coming in to the room?

This depends on your security protocols. Some organizations may decide  to terminate the exam and require another attempt. In some cases, if it seems an honest mistake, the organization may decide that the proctor can use discretion to permit the exam to continue.

Which is more secure, online or face-to-face proctoring?online proctoring

On balance, they are about equally secure.

Unfortunately there has been a lot of corruption with face-to-face proctoring, and online proctoring makes it much harder for participant and proctor to collude as there is no direct contact, and all communication can be logged.

But if the proctors are honest, it is easier to detect cheating aids in a face-to-face environment than via a video link.

What kind of exams is online proctoring good for?

Online proctoring works well for exams where:

  • The stakes are high and so you need the security of a proctor
  • Participants are in many different places, making travel to test centers costly
  • Participants are computer literate – have and know how to use their own PCs
  • Exams take 2-3 hours or less

If your technology or subject area changes frequently, then online proctoring is particularly good because you can easily give more frequent exams, without requiring candidates to travel.

What kind of exams is online proctoring less good for?

Online proctoring is less appropriate for exams where:

  • Exams are long and participants needs breaks
  • Exams where participants are local and it’s easy to get them into one place to take the exam
  • Participants do not have access to their own PC and/or are not computer literate

How do you prepare for online proctoring?

Here are some preparation tasks:

  • Brief and communicate with your participants about online proctoring
  • Define clearly the computer requirements for participants
  • Agree what happens in the event of incidents – e.g. suspected cheating, exam interruptions
  • Agree what ID is acceptable for participants and whether ID information is going to be stored
  • Make a candidate agreement or honor code which sets out what you expect from people to encourage them to take the exam fairly

I hope these Q&A and the linked presentation are interesting. You can find out more about Questionmark’s online proctoring solution here.

Job Task Analysis Surveys Legally Required?

John Kleeman Headshot

Posted by John Kleeman

I had a lot of positive feedback on my blog post Making your Assessment Valid: 5 Tips from Miami. There is a lot of interest in how to ensure your assessment is valid, ensuring that it measures what it is supposed to measure.

If you are assessing for competence in a job role or for promotion into a job role, one critical step in making your assessment valid is to have a good, current analysis of what knowledge, skills and abilities are needed to do the job role. This is called a job task analysis (JTA), and the most common way of doing this analysis is to conduct a JTA Survey.

Job Task Analysis SurveyIn a JTA Survey, you ask existing people in the job role, or other experts, what tasks they do. A common practice is to survey them on how important each task is, how difficult it is and how often it is done. The resultant reports then guide the construction of the test blueprint and which topics and how many questions on each you include in the test.

If you cannot show that your assessment matches the requirements of a job, then your assessment is not only invalid but it is likely unfair — if you use it to select people for the job or measure competence in the job. And if you use an invalid assessment to select people for promotion or recruitment into the job, you may face legal action from people you reject.

Not only is this common sense, but it was also confirmed by a recent US district court ruling against the Boston Police Department. In this court case, sergeants who had been rejected for promotion to lieutenant following an exam sued that the assessment was unfair, and won.

The judge ruled that the exam was not sufficiently valid, because it omitted many job skills crucial for a police lieutenant role, and so it was not fair to be used to select for the role (see news report).

The 82-page judge’s ruling sets out in detail why the exam was unfair. He references the Uniform Guidelines on Employee Selection Procedures which state:

“There should be a job analysis which includes an analysis of the important work behavior(s) required for successful performance and their relative importance”

But the judge ruled that although a job analysis had been done, it had not been used properly in the test construction process. He said:

“When using a multiple choice exam, the developer must convert the job analysis result into a test plan to ensure a direct and strong relationship between the job analysis and the exam.

However, in this case, the job analysis was not used sufficiently well to construct the exam. The judge went on to say:

The Court cannot find, however, that the test plan ensured a strong relationship between the job analysis and the exam. … too many skills and abilities were missing from the … test outline. 

Crucially, he concluded:

“And a high score on the … exam simply was not a good indicator that a candidate would be a good lieutenant”.

Due to the pace of business change and technological advance, job roles are changing fast. Make sure that you conduct regular JTAs  of roles in your organization and make sure your assessments match the most important job tasks. Find out more about Job Task Analysis here.

Making your Assessment Valid: 5 Tips from Miami

John Kleeman Headshot

Posted by John Kleeman

A key reason people use Questionmark’s assessment management system is that it helps you make more valid assessments. To remind you, a valid assessment is one that genuinely measures what it is supposed to measure. Having an effective process to ensure your assessments are valid, reliable and trustable was an important topic at Questionmark Conference 2016 in Miami last week. Here is some advice I heard:

Reporting back from 3 days of learning and networking at Questionmark Conference 2016 in Miami

Tip 1: Everything starts from the purpose of your assessment. Define this clearly and document it well. A purpose that is not well defined or that does not align with the needs of your organization will result in a poor test. It is useful to have a formal process to kick off  a new assessment to ensure the purpose is defined clearly and is aligned with business needs.

Tip 2: A Job Task Analysis survey is a great way of defining the topics/objectives for new-hire training assessments. One presenter at the conference sent out a survey to the top performing 50 percent of employees in a job role and asked questions on a series of potential job tasks. For each job task, he asked how difficult it is (complexity), how important it is (priority) and how often it is done (frequency). He then used the survey results to define the structure of knowledge assessments for new hires to ensure they aligned with needed job skills.

Tip 3: The best way to ensure that a workplace assessment starts and remains valid is continual involvement with Subject Matter Experts (SMEs). They help you ensure that the content of the assessment matches the content needed for the job and ensure this stays the case as the job changes. It’s worth investing in training your SMEs in item writing and item review. Foster a collaborative environment and build their confidence.

Tip 4: Allow your participants (test-takers) to feed back into the process. This will give you useful feedback to improve the questions and the validity of the assessment. It’s also an important part of being transparent and open in your assessment programme, which is useful because people are less likely to cheat if they feel that the process is well-intentioned. They are also less likely to complain about the results being unfair. For example it’s useful to write an internal blog explaining why and how you create the assessments and encourage feedback.

Lunch with a view at Questionmark Conference 2016 in Miami

Tip 5: As the item bank grows and as your assessment programme becomes more successful, make sure to manage the item bank and review items. Retire items that are no longer relevant or when they have been overexposed. This keeps the item bank useful, accurate and valid.

There was lots more at the conference – excitement that Questionmark NextGen authoring is finally here, a live demo of our new easy to use Printing and Scanning solution … and having lunch on the hotel terrace in the beautiful Miami spring sunshine – with Questionmark branded sunglasses to keep cool.

There was a lot of buzz at the conference about documenting your assessment decisions and making sure your assessments validly measure job competence. There is increasing understanding that assessment is a process not a project, and also that to be used to measure competence or to select for a job role, an assessment must cover all important job tasks.

I hope these tips on making assessments valid are helpful. Click here for more information on Questionmark’s assessment management system.

Next Page »