Making your Assessment Valid: 5 Tips from Miami

John Kleeman Headshot

Posted by John Kleeman

A key reason people use Questionmark’s assessment management system is that it helps you make more valid assessments. To remind you, a valid assessment is one that genuinely measures what it is supposed to measure. Having an effective process to ensure your assessments are valid, reliable and trustable was an important topic at Questionmark Conference 2016 in Miami last week. Here is some advice I heard:

Reporting back from 3 days of learning and networking at Questionmark Conference 2016 in Miami

Tip 1: Everything starts from the purpose of your assessment. Define this clearly and document it well. A purpose that is not well defined or that does not align with the needs of your organization will result in a poor test. It is useful to have a formal process to kick off  a new assessment to ensure the purpose is defined clearly and is aligned with business needs.

Tip 2: A Job Task Analysis survey is a great way of defining the topics/objectives for new-hire training assessments. One presenter at the conference sent out a survey to the top performing 50 percent of employees in a job role and asked questions on a series of potential job tasks. For each job task, he asked how difficult it is (complexity), how important it is (priority) and how often it is done (frequency). He then used the survey results to define the structure of knowledge assessments for new hires to ensure they aligned with needed job skills.

Tip 3: The best way to ensure that a workplace assessment starts and remains valid is continual involvement with Subject Matter Experts (SMEs). They help you ensure that the content of the assessment matches the content needed for the job and ensure this stays the case as the job changes. It’s worth investing in training your SMEs in item writing and item review. Foster a collaborative environment and build their confidence.

Tip 4: Allow your participants (test-takers) to feed back into the process. This will give you useful feedback to improve the questions and the validity of the assessment. It’s also an important part of being transparent and open in your assessment programme, which is useful because people are less likely to cheat if they feel that the process is well-intentioned. They are also less likely to complain about the results being unfair. For example it’s useful to write an internal blog explaining why and how you create the assessments and encourage feedback.

Lunch with a view at Questionmark Conference 2016 in Miami

Tip 5: As the item bank grows and as your assessment programme becomes more successful, make sure to manage the item bank and review items. Retire items that are no longer relevant or when they have been overexposed. This keeps the item bank useful, accurate and valid.

There was lots more at the conference – excitement that Questionmark NextGen authoring is finally here, a live demo of our new easy to use Printing and Scanning solution … and having lunch on the hotel terrace in the beautiful Miami spring sunshine – with Questionmark branded sunglasses to keep cool.

There was a lot of buzz at the conference about documenting your assessment decisions and making sure your assessments validly measure job competence. There is increasing understanding that assessment is a process not a project, and also that to be used to measure competence or to select for a job role, an assessment must cover all important job tasks.

I hope these tips on making assessments valid are helpful. Click here for more information on Questionmark’s assessment management system.

Single sign-on: secure and easy access

Bart Hendrickx Small

Posted by Bart Hendrickx

It’s Tuesday morning. You have just started your computer. Now it’s time to open your day-to-day tools: email, chat/phone, tasks and so on. As you go through your tasks, you realize you need to take that data security test you’ve been postponing.

Every day, you interact with various applications. Some are installed on your personal computer, others on servers managed either by your organization by vendors.  Your applications might come from a multitude of service providers, in-house or on the Cloud.

For many of those applications, you need to authenticate—to tell the applications who you are—so that the app can present the information that pertains to you. Sometimes this happens automatically. When your email client connects to the mail server, you read your emails, not those of your co-workers. Your email client has authenticated you against your mail server because you entered a username or email address, a password and some other data, way back when.

You often need to use different sign-ins for different apps. When you log in tor you Questionmark OnDemand portal, for instance, you enter a different username and password than the one you used to unlock your computer earlier today, (Your organization’s data security policy does not allow you to  store your organizational password in other systems.)

Want to learn more? I’ll be discussing this topic and more at the Questionmark Conference 2016 in Miami, April 12-15. Register before March 3 to take advantage of our final early-bird discounts.

Problem: Unrecognized username or password

You’ve logged in for your exam, but you get an error message. Maybe you mistyped the password? Second try. Nope; same results. You must have forgotten your password. You start an instant message window with your internal IT help desk. “Sorry, we don’t manage Questionmark OnDemand. Can you use its password reset function?”

You go back to your Questionmark login page, get a secure on-time login and establish a new, permanent password that complies with the data security policy — “I better not forget my password this time,” you say to yourself as you finally start your data security test. “Isn’t there something more convenient?”

Solution: Single sign-on

We all find ourselves in similar situations, but with Single sign-on (SSO) we can avoid them.

Since, there are several definitions of SSO, here’s how I’ll define it in the context of this blog:

Single Sign-On (SSO) for software is the ability for one application, the identity provider, to tell another application, the service provider, who you are.

By identity provider, I mean a system that contains digital identity information—also known as people data—on users, For example, think of social network sites or Active Directory from Microsoft.

The service provider is the system that users work with to do something—say Questionmark OnDemand, in the case of your data security test.

With SSO, a user does not log on directly to the service provider. Instead, they log on to an identity provider, which then tells the service provider who the user is. The identity provider and service provider have been configured to trust each other. So when the identity provider says: “This is Jane Doe,” the service provider will trust and accept that.

It is important to note that SSO is therefore not about creating accounts with the same usernames and passwords—a prevalent mechanism for different service providers. SSO is about making those service providers accept what an identity provider says about a user.

Why SSO?

SSO comes with several advantages. Users can access all applications that are linked to their identity providers—using one username and password for multiple systems. Depending on the capabilities of the applications and how things have been set up, the authentication can be seamless. You might log on to your identity provider when you start your computer, and the other applications (service providers) you access during the day will automatically check with your identity provider without you having to enter your username and password again.

SSO makes password management easier for IT administrators. Having an employee leave an organization might mean having to decommission access to dozens of service providers. If the authentication to those service providers has been set up with SSO, then an IT administrator only needs to decommission the employee’s identity provider account. Without that account, the employee can no longer log on to any of the linked applications.

There is one disadvantage to SSO: If the account at the identity provider is hacked, all linked applications can be compromised. It is therefore imperative the account is properly secured. How can you set up SSO to ensure its security and effectiveness? Watch for more posts on this subject, which will include information about our newly added support for a popular technique called SAML.

If you would like to learn more, attend my session: Secure Authentication: Accessing Questionmark OnDemand with SSO at the Questionmark Conference 2016, April 12-15. Register before March 3 to take advantage of our final early-bird discounts.

9 trends in compliance learning, training and assessment

John Kleeman HeadshotThis version is a re-post of a popular blog by John Kleeman

Where is the world of compliance training, learning and assessment going?

I’ve collaborated recently with two SAP experts, Thomas Jenewein of SAP and Simone Buchwald of EPI-USE, to write a white paper on “How to do it right – Learning, Training and Assessments in Regulatory Compliance[Free with registration]. In it, we suggested 9 key trends in the area. Here is a summary of the trends we see:

1. Increasing interest in predictive or forward-looking measures

Many compliance measures (for example, results of internal audits or training completion rates) are backwards looking. They tell you what happened in the past but don’t tell you about the problems to come. Companies can see clearly what is in their rear-view mirror, but the picture ahead of them is rainy and unclear. There are a lot of ways to use learning and assessment data to predict and look forward, and this is a key way to add business value.

2. Monitoring employee compliance with policies

A recent survey of chief compliance officers suggested that their biggest operational issue is monitoring employee compliance with policies, with over half of organizations raising this as a concern. An increasing focus for many companies is going to be how they can use training and assessments to check understanding of policies and to monitor compliance.

3. Increasing use of observational assessments

Picture of observational assessment on smartphoneWe expect growing use of observational assessments to help confirm that employees are following policies and procedures and to help assess practical skills. Readers of this blog will no doubt be familiar with the concept. If not, see Observational Assessments—why and how.

4. Compliance training conducted on mobile devices

The world is moving to mobile devices and this of course includes compliance training and assessment.

5. Informal learning

You would be surprised not to see informal learning in our list of trends. Increasingly we are all understanding that formal learning is the tip of the iceberg and that most learning is informal and often on the job.

6. Learning in the extended enterprise

Organizations are becoming more interlinked, and another important trend is the expansion of learning to the extended enterprise, such as contractors or partners. Whether for data security, product knowledge, anti-bribery or a host of other regulatory compliance reasons, it’s becoming crucial to be able to deliver learning and to assess not only your employees but those of other organizations who work closely with you.

7. Cloud

There is a steady movement towards the cloud and SaaS for compliance learning, training, and assessment – with the huge advantage of delegating all of the IT to an outside party being the strongest compelling factor.  Especially for compliance functions, the cloud offers a very flexible way to manage learning and assessment without requiring complex integrations or alignments with a company’s training departments or related functions.

8. Changing workforce needs

The workforce is constantly changing, and many “digital natives” are now joining organizations. To meet the needs of such workers, we’re increasingly seeing “gamification” in compliance training to help motivate and connect with employees. And the entire workforce is now accustomed to seeing high-quality user interfaces in consumer Web sites and expects the same in their corporate systems.

9. Big Data

E-learning and assessments are a unique way of touching all your employees. There is huge potential in using analytics based on learning and assessment data. We have the potential to combine Big Data available from valid and reliable learning assessments with data from finance, sales, and HR sources.  See for example the illustration below from SAP BusinessObjects showing assessment data graphed against performance data as an illustration of what can be done.

data exported using OData from Questionmark into SAP BusinessObjects

For information on these trends, see the white paper written with SAP and EPI-USE: “How to do it right – Learning, Training and Assessments in Regulatory Compliance”, available free to download with registration.

If you have other suggestions for trends, feel free to contribute them below.

Questionmark Secure patent granted

Copy of patent grant imageJohn Kleeman HeadshotPosted by John Kleeman

I’m pleased to let you know that Questionmark has been granted a US patent for one of our innovations in our secure browser, Questionmark Secure.

Questionmark was one of the pioneers in secure browsers. A secure or lock-down browser is designed to help organizations provide a secure environment in which to deliver higher stakes assessments such as tests and exams. It helps prevent cheating in an assessment by disabling functions that participants could use to print or copy exam material, “accidentally” exiting a test, or gaining access to materials on their computers or the Internet that could give an unfair advantage.

Here’s a little history on how we got here:

Our first secure browser called Perception Secure Browser was produced in 1999 – you can see the press release here.

This browser, like many current secure browsers, was started up to run a specific test. But many of our customers requested something slightly different – they wanted a participant to be able to use an ordinary web browser to participate in learning courses, navigate through registration screens and/or use a learning management system, and have the secure browser launched automatically once the assessment starts. This would allow the participant to use standard browser capabilities whilst learning or registering – but when security becomes important, have the secure browser take over. Then when the assessment is over, the participant can revert to the normal browser.

Questionmark Secure splash screenThis required some clever technical work to make happen, but in 2003, we introduced  a new secure browser called Questionmark Secure which did exactly this (you can see the press release here). A participant can use a normal browser to navigate through learning or registration screens, and when they reach the secure assessment, Questionmark Secure takes over to make the assessment process secure. Our current Questionmark Secure product, though hugely improved over the 2003 version (!) uses the same concept – originally invented by Eric Shepherd, Paul Roberts and myself.

Because Questionmark Secure used innovatory technology, we filed for a patent on some of the methods and technology used and related to it . It took over 10 years to be granted but Questionmark is proud to have received U,S. Patent Number 9,055,048 recently for this unique invention.  The abstract for the patent reads:

A method for interacting with a user, comprising communicating with at least one cooperative server through a normal browser; automatically receiving encrypted data having an associated received type code indicative of a requirement for a secure browser having restricted functionality with respect to a functionality of the normal browser; selectively and automatically invoking the secure browser for handling of the received encrypted data based on the received type code associated with the received encrypted data; receiving the encrypted data with the invoked secure browser for handling thereof, wherein the received encrypted data is not available for use by the user in the normal browser and the invoked secure browser imposes restrictions on availability outside of the secure browser of decrypted data derived from the encrypted data; and communicating an input from the user, through the secure browser, to the at least one cooperative server.

For Questionmark customers, this patent helps reinforce Questionmark’s role as a leader in computerized assessment.

For more information on Questionmark Secure, see https://www.questionmark.com/content/questionmark-secure.

SAP to present their global certification program at London briefing

Chloe MendoncaPosted by Chloe Mendonca

A key to SAP’s success is ensuring that the professional learning path of skilled SAP practitioners is continually supported – thereby making qualified experts on their cloud solutions readily available to customers, partners and consultants.

In a world where current knowledge and skills are more important than ever, SAP needed a way to verify that their cloud consultants around the world were keeping their knowledge and skills up-to-date  with rapidly changing technology. A representative of the certification program at SAP comments:breakfast briefing

It became clear that a certification that lasted for two or three years didn’t cut it any longer – in all areas of the portfolio. Everything is evolving so quickly, and SAP has to always support current, validated knowledge.”

Best Practices from SAP

The move to the cloud required some fundamental changes to SAP’s existing certification program. What challenges did they face? What technologies are they using to ensure the security of the program? Join us on the 21st of October for a breakfast briefing in London, where Ralf Kirchgaessner, Manager of Global Certification at SAP, will discuss the answers to these questions. Ralf will tell how the SAP team planned for the program, explain its benefits and share lessons learned.

Click here to learn more and register for this complimentary breakfast briefing *Seats are limited

High-Stakes Assessments

The briefing will  include a best-practice seminar on the types of technologies and techniques to consider using as part of your assessment program to securely create, deliver and report on high-stakes tests around the world. It will highlight technologies such as online invigilation, secure browsers and item banking tools that alleviate the testing centre burden and allow organisations and test publishers to securely administer trustable tests and exams and protect valuable assessment content.

What’s a breakfast briefing?

You can expect a morning of networking, best practice tips and live demonstrations of the newest assessment technologies.The event will include a complimentary breakfast at 8:45 a.m. followed by presentations and discussions until about 12:30 p.m.

Who should attend?

These gatherings are ideal for people involved in certification, compliance and/or risk management, and learning and development.

When? Where?

Wednesday 21st October at Microsoft’s Office in London, Victoria — 8:45 a.m. – 12:30 p.m

Click here to learn more and register to attend

Agree or disagree? 10 tips for better surveys — part 3

John Kleeman HeadshotPosted by John Kleeman

This is the third and last post in my “Agree or disagree” series on writing effective attitude surveys. In the first post I explained the process survey participants go through when answering questions and the concept of satisficing – where some participants give what they think is a satisfactory answer rather than stretching themselves to give the best answer.

In the second post I shared these five tips based on research evidence on question and survey design.

Tip #1 – Avoid Agree/Disagree questions

Tip #2 – Avoid Yes/No and True/False questions

Tip #3 – Each question should address one attitude only

Tip #4 – Minimize the difficulty of answering each question

Tip #5 – Randomize the responses if order is not important

Here are five more:

Tip #6 –  Pretest your survey

Just as with tests and exams, you need to pretest or pilot your survey before it goes live. Participants may interpret questions differently than you intended. It’s important to get the language right so as to trigger in the participant the right judgement. Here are some good pre-testing methods:

  • Get a peer or expert to review the survey.
  • Pre-test with participants and measuring the response time for each question (shown in some Questionmark reports). A longer response time could be connected with a more confusing question.
  • Allow participants to provide comments on questions they think they are confusing.
  • Follow up with your pretesting group by asking them why they gave particular answers or asking them what they thought you meant by your  questions.

Tip #7 – Make survey participants realize how useful the survey is

The more motivated a participant is, the more likely he or she is to answer optimally rather than just satisficing and choosing a good enough answer. To quote Professor Krosnick in his paper The Impact of Satisficing on Survey Data Quality:

“Motivation to optimize is likely to be greater among respondents who think that the survey in which they are participating is important and/or useful”

Ensure that you communicate the goal of the survey and make participants feel that filling it in usefully will be a benefit to something they believe in or value.

Tip #8. Don’t include a “don’t know” option

Including a “don’t know” option usually does not improve the accuracy of your survey. In most cases it reduces it. To those of us used to the precision of testing and assessment, this is surprising.

Part of the reason is that providing a “don’t know” or “no opinion” option allows participants to disengage from your survey and so diminishes useful responses. Also,  people are better at guessing or estimating than they think they are, so they will tend to choose an appropriate answer if they do not have an option of “don’t know”. See this paper by Mondak and Davis, which illustrates this in the political field.

Tip #9. Ask questions about the recent past only

The further back in time they are asked to remember, the less accurately participants will answer your questions. We all have a tendency to “telescope” the timing of events and imagine that things happened earlier or later than they did. If you can, ask about the last week or the last month, not about the last year or further back.

Picture of a trends graphTip #10 – Trends are good

Error can creep into survey results in many ways. Participants can misunderstand the question. They can fail to recall the right information. Their judgement can be influenced by social pressures. And they are limited by the choices available. But if you use the same questions over time with a similar population, you can be pretty sure that changes over time are meaningful.

For example, if you deliver an employee attitude survey with the same questions for two years running, then changes in the results to a question (if statistically significant) probably mean a change in employee attitudes. If you can use the same or similar questions over time and can identify trends or changes in results, such data can be very trustworthy.

I hope you’ve found this series of articles useful.  For more information on how Questionmark can help you create, deliver and report on surveys, see www.questionmark.com. I’ll also be presenting at Questionmark’s 2016 Conference: Shaping the Future of Assessment in Miami April 12-15. Check out the conference page for more information.