Eight ways to check if security is more than skin deep

Picture of computer and padlockJohn Kleeman HeadshotPosted by John Kleeman

The assessment industry has always been extremely careful about exam security and ways to prevent cheating. As cloud and online assessment takes over as delivery models, it’s critical we all deeply embed IT security in our culture to ensure that computer vulnerabilities don’t leak sensitive data or disrupt the integrity of the assessment process.

Many years ago, Questionmark realized that data protection and IT security were critical to our success. We re-formed our culture to make security a priority. We followed our own path and looked for opportunities to learn from others such as Bill Gates and his famous trustworthy computing memo, part of which is quoted below:

… when we face a choice between adding features and resolving security issues, we need to choose security. Our products should emphasize security right out of the box, and we must constantly refine and improve that security as threats evolve. …  These principles should apply at every stage of the development cycle of every kind of software we create …

Questionmark understands that we’re in an arms race. We stay vigilant and look for opportunities to improve our security. Here are eight key ways in which we have embedded security deep within our company. If you are an assessment provider, we’d encourage you to find your own way to follow suit. And if you are a customer, here are eight questions you can ask to identify whether an assessment provider is truly working to be as secure as it can,  instead of just claiming to be secure when in fact security is only skin deep.

1. Who does the security function report to?

At Questionmark our security officer reports directly to me as Questionmark Chairman. If security reports directly into IT or product development, a security concern might be overruled by operational need. We’ve found this separation very helpful to ensure security gets listened to throughout the organization.

2. Would a security flaw hold up a release?

In any sensible company, this has to be true. Feature improvements in software are important, but if there is a serious security issue, it needs to be fixed first. Developers need to know that they can’t make a release unless it is secure.

3. How do you check your employees know about security?

Questionmark trains all our employees on data security but how do we know they understand? We practice what we preach and everyone from senior management to sales to accounting to developers needs to take and pass a data security test every year to check understanding. I’d encourage everyone in the assessment industry to follow this approach.

4. How deep is your team’s knowledge of IT security?

SaaS security is complex. There are many layers to security and any weakness can lead to a vulnerability. Equally throwing resources in the wrong place won’t really help. We are fortunate to have at least half a dozen experts within Questionmark who have deep knowledge of and passion for different aspects of security. This helps us get things right,.

5. Is your ecosystem secure?

Every company operates in an ecosystem , and it’s the ecosystem that needs to be secure. Questionmark works with our suppliers, subcontractors and partners to help them to be secure, including offering training and advice. We even want our competitors to be secure as any breaches in the assessment industry would be hurtful to all.

6. How transparent and open are you on your security?

Security by obscurity is not secure. Questionmark shares information on the security of our OnDemand service in white papers (Security of Questionmark’s US OnDemand Service and Security of Questionmark’s EU OnDemand Service) and have “red papers” which describe our security and business continuity planning in detail, available under NDA to prospective customers. The review process as customers ask questions about these provides comfort for customers and input to us to improve our security.

7. What kinds of external review do you allow?

As we shared in Third-party audits verify our platform’s security, we run regular penetration tests by a third party company, Veracode on Questionmark OnDemand. We are also fortunate to have many customers who care deeply about security and undertake their own audits and reviews by experts. We welcome such review and learn from it to improve our own security.

8. Are you completely satisfied with your security?

imageAbsolutely not. There is an arms race happening in the security world. Hackers and other bad actors are increasing their capabilities and however good you are, if you rest on your laurels, the arms race will overtake you. See for example the graph to the right from Verizon showing the increase in breaches over time.

Questionmark, like other good SaaS companies, has a policy of continual improvement – we want to be much better each year than the last.

This video provides an overview of how Questionmark builds security into its products from day one. Watch below:

Conference Wrap Up: Tips, Advice & Pictures from Napa

Julie Delazyn HeadshotPosted by Julie Delazyn

flickrThe Questionmark Users Conference is the most important learning event of the year. With over a dozen sessions to attend and topics ranging from penetration testing to measuring and understanding your assessment results, there is so much knowledge packed within three days.

Assessment security was an important topic at the Conference in Napa. Questionmark Chairman John Kleeman took to the blog last week to lay out some security advice he heard from attendees. You can check out his blog post, Assessment Security: 5 Tips from Napa, to learn more.

Frequent blog contributor, Psychometrician and Reporting and Analytics Manager, Austin Fossey, presented a number of  sessions at the conference this year. According to Austin, “regardless of their individual roles or organizations’ goals, Questionmark users are first and foremost measurement professionals.” Impressed by our customers’ commitment to look for ways to always improve their measurements, validity, and impact for stakeholders, Austin wrote a blog post highlighting some of the example that struck him. You can read more about the stories he heard from our customers about their assessment programs in his blog post: 2015 Users Conference – A Gathering of Measurement Professionals.

If you did not have a chance to attend the Conference in Napa, there is always next year! Look out for dates and a special location announcement on the blog. For those of you who attended and would like to relive some of the special moments spent in Napa, you can check out the pictures now on our Flickr page. The photos highlight moments from the conference as well as from our special evening event at Markham Winery.

conf goers banner

Nine tips on recommended assessment practice — from Barcelona

John Kleeman HeadshotPosted by John Kleeman

Something I enjoy most about our users conferences is the chance to learn from experts about good practice in assessments. Most of our customers have deep knowledge and insightful practical experience, so there is always much to learn.

Here are some tips I picked up last week at our recent European Users Conference in Barcelona.Questionmark2013_DSC3209

1. Make sure to blueprint. It’s critical to have a detailed design (often called a blueprint) for an assessment – or as one user shared, “Without a blueprint, you don’t have an assessment”.

2. Network to get SMEs. With technology changing quickly, if your assessments assess IT or other new technology, the content changes very quickly and the quality of your subject matter experts (SMEs) who create and review items is critical. As an assessment owner, use networking skills to get the right SMEs on board; getting them engaged and building trust are essential.

3. Test above knowledge. Develop questions that test application or comprehension, for instance using scenarios. They are more likely to make your test valid than questions that simply test facts.

4. Give employees ownership of their own compliance testing. If employees have to take annual refresher tests, give them the responsibility to do so and encourage sel- learning and pre-reading. Give them plenty of time (e.g. 6 weeks’ warning), but make it their responsibility to take and pass the test in the window, not yours to keep on reminding them.

5. Gather feedback from participants. Make sure you solicit feedback from your participants on tests and the testing experience. That way you will learn about weak questions and how to improve your testing process. And you also make participants feel that the process is fairer.

6. Use job/task analysis. Asking questions about jobs and tasks is the best way to specify the criteria used to judge competency or proficiency. These questions can be automated in Questionmark right now. Watch this space for improvements coming to make this easier.

7. Look at Questionmark Live for item review workshops. If you have any informal or informal process for having groups of people working on or reviewing items, look at Questionmark Live. It’s free to use, has great group working capability and improves productivity. A lot of organizations are having success with it.

8. Keep feedback short and to the point… especially on mobile devices where people won’t read long messages.Questionmark2013_DSC3215

9. Look for live data, not just your rear view mirror. Data is important – without measurement we cannot improve. But make sure the data you are looking at is not dead data. Looking into the rear view mirror of what happened in the past doesn’t help as much as using reports and analytics from Questionmark to discover what is happening now, and use that data to improve things.

I hope some of these tips can help you in your work with assessments.

I will write in the spring about the tips I gather at the 2014 U.S. Users Conference in San Antonio!

10 Reasons for Using an Assessment Management System for Compliance

Posted by John Kleeman

Most LMSs (learning management systems) have the capability to deliver basic quizzes and surveys. So is an LMS good enough to deliver online compliance assessments? Or do you need an assessment management system?

A strength of LMSs is that they roll up all training, for example face-to-face classroom events, and they’re often used as a system of record for compliance training events. But many companies that are professional about their use of assessments in compliance use an assessment management system as well as (or sometimes instead of) an LMS. Here are 10 of the reasons I hear for doing this.

Observational assessment

1. A key trend in compliance is to measure behaviour, not just knowledge. A great way to do this is observational assessments, during which an observer watches someone do something (e.g. interview a customer, use a machine) and rates them on an iPad or smartphone.  The ability to deliver assessments in many different environments is a leading advantage of a comprehensive assessment management system.

2. Running a professional assessment programme needs an item bank, where all your questions are organized by topic and metadata, so you can re-use questions and easily review and update them. Many LMSs link questions and assessments to courses, but you need a searchable item bank once you get a certain volume of assessments.

3. Assessment management systems usually provide an easier and more friendly user interface for Subject Matter Experts (SMEs) to author questions, for instance our easy-to-use Questionmark Live collaborative authoring environment.

Create Question set, add questions to set, download or email questions in a qpack, import into Perception

4. As mentioned in my earlier blog post, How Topic Feedback can give Compliance Assessments Business Value, being able to score and give feedback at the topic level lets you provide actionable feedback in compliance. You don’t just know  people are weak, you know where they are weak and how to improve it.

5. Assessment management systems offer more question types, allowing more variety and more engaging and realistic questions.

6. An assessment management system like Questionmark lets you deliver assessments on paper as well as on-screen, and also on mobile devices including smartphones and iPads; you can deliver assessments in more places.

Test analysis report7. Most LMSs have only basic assessment reporting – but to make your assessments valid and reliable and legally defensible, you need item and test statistics reports and other professional reports.

8. Assessments can continue to be delivered even if you change LMS – and many organizations are thinking of changing LMS or moving the LMS to the cloud.

9. Often an LMS is provisioned only for employees, but you may need to assess partners or contractors; it’s easy to allow direct login to Questionmark if desired.

Questionmark Secure icon10. Last but not least, the typical LMS does not major in test security. Most employees taking compliance assessments will not want to cheat, but it’s useful to have the stronger security — allowing monitoring, preventing cheating and avoiding fraud — of a professional assessment management system.

Bottom line, an assessment management system gives you trustable results that you can rely on. A large organization relies on employees who are geographically and functionally separated, but they all must act competently and follow proper procedures to make the business run effectively. Assessments delivered online to employees via an assessment management system are one of the few ways and likely the best way to touch individually your entire workforce and ensure that they understand their role in your business and what is required of them to meet business and regulatory needs.

Ten top tips for effective online assessment from HSBC’s Matt Bushby

Posted by John Kleeman

HSBC – a multinational banking and financial services organisation – uses Questionmark Perception worldwide for delivering computer-based, online competency tests for employees to check they understand products and regulations. Matt Bushby is an Assessment Specialist and the Global Subject Matter Expert for Questionmark Perception at HSBC.

Matt has been responsible for the transfer of paper-based testing to online assessment in the UK since 2006 and has supported and advised a dozen or so other HSBC country teams in their move from different tools and systems to a joined-up, single assessment system from Questionmark. By using assessments in this way, HSBC are able to meet strict financial regulations in each country and also ensure a premium service to customers.

Matt Bushby and his lovely children!

I asked him to share from his experience some key tips for making online assessment effective within a large company, and here is his advice:

1. Start with “why” you are assessing. You need to have an informed conversation within the business to get discussion and buy-in. Some typical questions which can stimulate the conversation:

· How many people are likely to take the assessment?

· What are the consequences for failing?

· How high or low stakes is the assessment?

· Can people re-take the test?

· What follow-on learning will you use for people who pass the test but get some questions wrong?

2. Work out some key policy questions for the assessment:

· Should it be open-book or closed-book?

· Should everyone see the same questions or should there be random selection?

· Should there be a time limit or should it be untimed?

· Do participants have to answer all questions or can they leave some blank?

· Should the assessment give feedback on wrong answers?

3. Plan the assessment before you start writing the questions. Work out the topic granularity and structure and how many questions you need in each topic. For example, a typical assessment will call in questions from many different topics and for many assessments. HSBC typically create 3 spare questions for each question actually delivered to a test-taker.

4. Agree with the business how the structure of the assessment is going to be validated and agreed, both initially and on-going as the business shifts focus over time. Typically this would be by the business reviewing the overview blueprint for an assessment.

5. HSBC finds that multiple choice questions work well for much competency assessment, providing they are written well. It’s key to provide guidance to subject matter experts in question design. Here are some key points :

· have a consistent number of choices,

· make all incorrect choices reasonable,

· avoid negative language,

· don’t make the correct choice the longest!

6. When it meets the business need, encourage scenario questions that test application rather than factual questions. Scenarios can work well as multiple choice questions, where distractors describe common mistakes. You may need to allow longer time limits for assessments with scenario questions.

7. Be careful about using non-accessible visual elements like screenshot images in questions, as any large company will likely have visually impaired employees or others with accessibility needs that need to be catered for. It’s usually best to have one assessment that all can take, rather than having to make variants.

8. Matt finds giving generic feedback to read up on a topic or syllabus area very effective, as this means that he doesn’t need to include the original question in the feedback. This also means less risk of leakage via sharing feedback between participants.

9. For assessments that are always live (for example assessments as part of on-boarding training), keep in touch with subject matter experts and the business to ensure that these are regularly reviewed and stay up to date.

10. When introducing a new assessment, follow the approach of pilot / review / amend / launch.