Predicting Success at Entel Begins with Trustworthy Assessment Results [Case Study]

Julie ProfilePosted by Julie Delazyn

Entel is one of Chile’s largest telecommunications firms, serving both the consumer and commercial sectors. With more than 12,000 employees across its extended Entelenterprise, Entel provides a broad range of mobile and fixed communications services, IT and call center outsourcing and network infrastructure services.

The Challenge

Success in the highly competitive mobile and telecommunications market takes more than world-class infrastructure, great connectivity, an established dealer network and extensive range of retail location. Achieving optimal on-the-job performance yields a competitive edge in the form of satisfied customers, increased revenues and lower costs. Yet actually accomplishing this objective is no small feat – especially for an industry job role notorious for high turnover rates.

With these challenges in mind, Entel embarked on an innovative new strategy to enhance the predictability of the hiring, onboarding, training and developing practices for its nationwide team of 6,000+ retail store and call center representatives.

Certification as a Predictive Metric

Entel conducted an exhaustive analysis – a “big data” initiative that mapped correlations between dozens of disparate data points mined from various business systems, HR systems as well as assessment results – to develop a comprehensive model of the factors contributing to employee performance.

Working with Questionmark OnDemand enabled Entel to create the valid and reliable tests and exams necessary to measure and document representatives’ knowledge, skills and abilities.

Find out more about Entel’s program planning and development, which helped identify and set benchmarks for required knowledge and skills, optimal behaviors and performance metrics, its use of SAP SuccessFactors to manage and monitor performance against many of the key behavioral aspects of the program, as well as the growing role their trustworthy assessment results are having on future product launches and the business as a whole.

Click here to read the full case study.

Security, Reporting and Online Testing in Academia: A Q&A with Don Kassner

Julie ProfileI recently spoke with Don Kassner, who has joined Questionmark as vice president of academic markets. I wanted to take a moment to welcome Don and to ask him a few questions about his extensive background, his new job role and his hopes for the future. Here’s a snippet of our conversation:

Don Kassner, VP of academic markets, Questionmark

You have had an extensive career. What is your background, and how has it influenced the insight you have on the learning and assessment world?

At heart, I am an entrepreneur.  I started my first business when I was 8 years old and have been starting and building things ever since. In the last 15 years or so I have focused on technology, training and education. During that time, I have served as the CFO for an auto dealership training company and as president of a small university in addition to founding and building the largest online proctoring company. I have also served on the faculty of San Jose State University (economics) and have been an active member of multiple onsite accreditation evaluation teams.

Now that you are part of the Questionmark team, can you share a little about your new role and the goals you have for this position?

I am excited to reach out to the academic markets and address ongoing concerns around testing.

Colleges and universities want to deliver tests and exams that are consistent, fair, reliable and defensible. They want to deliver and monitor course evaluations, identify knowledge gaps and place students in appropriate courses, and be able to analyze questions to determine which ones are valid and fair. They want to do this while increasing student satisfaction and reducing cheating.

With Questionmark’s extensive security, reporting and analytics tools, that’s all possible.  My focus is to leverage my insight and experience and put this secure and easy-to-use tool in the right hands.

Can you share some of the important topics surrounding the academic markets that are on your radar?

In academia, there are real issues related to online testing. These are the top two:

  • Is the student doing the work? – With technology comes flexibility and as the stakes become higher, students will naturally look for ways to enhance their scores.
  • Are the exam results fair?  Did all the students face the same conditions with the same opportunities and were the exam rules followed?

Questionmark OnDemand, especially its analytics and reporting tools, can provide reports that can uncover the meaning hidden within assessment results, such as an item or assessment’s reliability and defensibility. In regards to security, Questionmark Secure’s lock-down browser plays a huge role in helping organizations provide a secure environment in which to deliver high stakes assessments. This can significantly help reduce the risk of cheating when deployed along with other defenses to combat impersonation and content theft.

We’re very excited to have you on board! What are your plans for keeping in touch with us on The Questionmark Blog?

Once a teacher, always a teacher! I look forward to turning back the clock to my university days and speaking on the hot-button topics that surround academic testing. My experience in education and entrepreneurship will help me as I share my thoughts on current market trends and the future of assessment and testing.

Thank you Don for taking the time to speak to us today!

You can follow Don on Twitter and connect with him on LinkedIn

SAML 101

Bart Hendrickx SmallPosted by Bart Hendrickx

As I mentioned in my previous post on SSO, Single Sign-On: Who’s Involved?, we’ll take a look at SAML to understand what it is and how it’s used with SSO. In this post I’ll explain what SAML is, and I will offer an example use case in my next post.

Webinar

So, What Is SAML?

SAML, or Security Assertion Markup Language, is a protocol that allows systems to exchange authentication data on users. (It facilitates other use cases as well, but I will focus on authentication.) What does that mean? It means that one system can ask: “Who is this user?” and another system can answer: “This is Jane Doe.” As I mentioned in my post previous post, I am talking about the service provider (SP) and identity provider (IdP) respectively.

Service providers (SP) in this context can be any software system with which you can do something, such as sending and receiving email, tracking projects or delivering assessments. Similarly, an identity provider (IdP) can be any software system that contain data on users that you can use to determine who those users are.

If you manage an SP, you probably don’t want just any IdP telling you who someone is. You will typically trust only one or a few IdPs. And if you are in charge of an IdP, you will likewise prefer to send user data only to those SPs you know and trust. To accomplish that, the SP and IdP exchange data allowing them to establishing a trust relationship. Those data are often called federation metadata, federation referring to the fact that there is an alliance between the different systems.

SAML is a popular protocol to set up such federations between service providers and identity providers. Look up SAML in your favorite search engine and you will get many results. One of its advantages is that it is extensible, meaning that you can exchange information that is relevant to your situation. For example, do you have an IdP that stores the hire date for an employee (or enrollment date of a student)? Do you want to share those data with an SP so that it can decide whether the user is allowed to access a certain resource? Then you can set up the federation in such a way that the IdP will send an attribute for hire (or enrollment) date to the SP.

Another advantage, and it is a huge one, is that SAML can be used in situations where the IdP and SP cannot talk to each other, for example because they are on different networks. You may have an IdP running on your internal network, behind a firewall. Your SP may be available in the cloud, as is the case with Questionmark OnDemand. The SP cannot talk to the IdP because it cannot “see” it. However, that’s not a problem for SAML. In my next post, we’ll take a look at a typical use case so we can see the practicality of using SAML with SSO.

 

6 Steps to Authoring Trustworthy Assessments

AprilPosted by April Barnum

I recently met with customers and the topic of authoring trustworthy assessments and getting back trustable results was a top concern. No matter what they were assessing on, everyone wants results that are trustable, meaning that they are both valid and reliable. The reasons were similar, with the top three being: Safety concerns, being able to assert job competency, and regulatory compliance. I often share this white paper: 5 steps to better tests, as a strong resource to help you plan a strong assessment, and I encourage you to check it out. But here are six authoring steps to that can help you achieve trustworthy assessment results:

  1. Planning the assessment or blueprinting it. You basically are working out what it is that the test covers.
  2. Authoring or creating the items.
  3. Assembling the assessment or harvesting the items and assemble them for use in a test.
  4. Piloting and reviewing the assessment prior to using it for production use.
  5. Delivering the assessment or making the assessment available to participants; following security, proctoring and other requirements set out in the planning stage.
  6. Analyzing the results of the assessment or looking at the results and sharing them with stakeholders. This step also involves using the data to weed out any problem items or other issues that might be uncovered.

Each step contributes to the next, and useful analysis of the results is only possible if every previous stage has been done effectively. In future posts, I will go into each step in detail and highlight aspects you should be considering at each stage of the process.

assessment plan

Single Sign-On: Who’s Involved?

Bart Hendrickx SmallPosted by Bart Hendrickx

This is the third post in a series on single sign-on (SSO). Go here for posts 1 and 2:

  1. Single sign-on: secure and easy access
  2. Single Sign-On Pros and Cons

In my first post, I offered this definition for SSO:

Single Sign-On (SSO) for software is the ability for one application, the identity provider, to tell another application, the service provider, who you are.

As you can gather from that definition, SSO involves two parties: the identity provider and the service provider. That is not the complete picture. In many cases, there is also the user and the user agent.

In this post, I explain who those parties are.

Service Provider (SP)

The service provider (SP) is the system a user wants to do something with. For example, Questionmark OnDemand: the user can be a participant who wants to take an assessment. Questionmark OnDemand provides the service of delivering an assessment that the user can take.

When single sign-on is set up, the SP relies on another system to authenticate a user. Therefore, the service provider is also called the relying party (RP). To keep things simple, I will stick to using “SP”.

Identity Provider (IdP)

You know that the SP relies on another system for authentication. That other system is called the identity provider (IdP). The IdP provides identity information on a user to the SP, so that the SP can make a decision to let the user in.

There are many systems that can act as identity providers and some systems specialize in identity. You will recognize many of these: Microsoft Active Directory, Google Account, Facebook, Twitter, LinkedIn—the list is very long.

User

There is nothing special to user: this is simply the person who wants to do something, such as the participant who wants to take an assessment. Well, the user does not need to be a person; it can also be a system. I will continue using the concept of “person” when I talk about “user”, for simplicity. (Did I already tell you I like to keep things simple?)

This party is sometimes called principal.

User Agent

The user agent is a piece of software, an application, which acts on behalf of the user. That sounds complicated, so let me give you an example. Your web browser is a user agent. When you visit a web page, your browser requests content on your behalf. Is an image referenced in the content? Your browser downloads it, on your behalf. Do you want to submit a form, such as an order with payment details? Your browser sends the information to the website, on your behalf.

In the context of SSO, the user agent often is a browser or an application that can access something on the Internet, such as an app on your smartphone to read emails or access a social network.

The Parties Working Together

Let me bring this together in a picture.

SSO2

You see a group of users, say, participants who want to take an assessment. They use their web browser, which is a user agent, to connect to the service provider; Questionmark OnDemand in this example.

The service provider relies on an identity provider to say who these users are. In this example, instead of talking directly to the identity provider, the service provider talks to the user agent (browser), who talks to the identity provider. “Who is this user?” the user agent asks the identity provider. The user may need to enter a username and password, on a page which is, you guessed it, displayed by the user agent.

When the identity provider successfully authenticates the user, it responds to the user agent: “This is Jane Doe.” The user agent passes that information on to the service provider, who then decides to give access: “This is Jane Doe and she is a participant; I will make this assessment available to her.”

How can the service provider and identity provider talk to each other, in this case indirectly through the user agent (browser)? They speak a shared language; a protocol that is used to exchange identity information. There are several popular such protocols. I will discuss them in a following post.

Develop Better Tests with Item Analysis [New eBook]

Posted by Chloe Mendonca

Item Analysis is probably the most important tool for increasing test effectiveness.  In order to write items that accurately and reliably measure what they’re intended to, you need to examine participant responses to each item. You can use this information to improve test items and identify unfair or biased items.

So what’s the process for conducting an item analysis? What should you be looking for? How do you determine if a question is “good enough”?

Questionmark has just published a new eBook “Item Analysis Analytics, which answers these questions. The eBook shares many examples of varying statistics that you may come across item analysis ebookin your own analyses.

Download this eBook to learn about these aspects of analytics:

  • the basics of classical test theory and item analysis
  • the process of conducting an item analysis
  • essential things to look for in a typical item analysis report
  • whether a question “makes the grade” in terms of psychometric quality

This eBook is available as a PDF and ePUB suitable for viewing on a variety of mobile devices and eReaders.

I hope you enjoy reading it!

Next Page »