4 Ways to Identify a Content Breach

Austin Fossey-42Posted by Austin Fossey

In my last post, I discussed five ways you can limit the use of breached content so that a person with unauthorized access to your test content will have limited opportunities to put that information to use; however, those measures only control the problem of a content breach. Our next goal is to identify when a content breach has occurred so that we can remedy the problem through changes to the assessment or disciplinary actions against the parties involved in the breach.

mitigating risk

Interested in learning about item analysis or how-to take your test planning to the next level? I will be presenting a series of workshops on at the Questionmark Conference 2016: Shaping the Future of Assessment in Miami, April 12-15. 

Channel for Reporting

In most cases, you (the assessment program staff) will not be the first to find out that content has been stolen. You are far more likely to learn about the problem through a tip from another participant or stakeholder. One of the best things your organization can do to identify a content breach is to have a clear process for letting people report these concerns, as well as a detailed policy for what to do if a breach is found.

For example, you may want to have a disciplinary policy to address the investigation process, potential consequences, and an appeals process for participants who allegedly gained unauthorized access to the content (even if they did not pass the assessment). You may want to have legal resources lined up to help address non-participant parties who may be sharing your assessment content illegally (e.g., so-called “brain dump” sites). Finally, you should have an internal plan in place for what you will do if content is breached. Do you have backup items that can be inserted in the form? Can you release an updated form ahead of your republishing schedule? Will your response be different depending on the extent of the breach?

Web Patrol Monitoring

Several companies offer a web patrol service that will search the internet for pages where your assessment content has been posted without permission. Some of these companies will even purchase unauthorized practice exams that claim to have your assessment content and look for item breaches within them. Some of Questionmark’s partners provide web patrol services.

Statistical Models

There are several publicly available statistical models that can be used to identify abnormalities in participants’ response patterns or matches between a response pattern and a known content breach, such as the key patterns posted on a brain-dump site. Several companies, including some of Questionmark’s partners, have developed their own statistical methods for identifying cases where a participant may have used breached content.

In their chapter in Educational Measurement (4th ed.), Allan Cohen and James Wollack explain that all of these models tend to explore whether the amount of similarity between two sets of responses can be explained by chance alone. For example, one could look for two participants who had similar responses, possibly suggesting collusion or indicating that one participant copied the other. One could also look for similarity between a participant’s responses and the keys given in a leaked assessment form. Models also exist for identifying patterns within groups, as might be the case when a teacher chooses to provide answers to an entire class.

These models are a sophisticated way to look for breaches in content, but they are not foolproof. None of them prove that a participant was cheating, though they can provide weighty statistical evidence. Cohen and Wollack warn that several of the most popular models have been shown to suffer from liberal or conservative Type I error rates, though new models continue to improve in this area.

Item Drift

When considering content breaches, you might also be interested in cases where an item appears to become easier (or harder) for everyone over time. Consider a situation where your participant population has global access to information that changes how they respond to an item. This could be for some unsavory reasons (e.g., a lot of people stole your content), or it could be something benign, like a newsworthy event that caused your population to learn more about content related to your assessment. In these cases, you might expect certain items to become easier for everyone in the population.

To detect whether an item is becoming easier over time, we do not use the p value from Classical Test Theory. Instead, we use Item Response Theory (IRT) and a Differential Item Functioning to detect item drift, which is changes in an item’s IRT parameters over time. This is done with Thissen, Steinberg, and Wainer’s likelihood ratio test that they detailed in Test Validity. Creators of IRT assessments use item parameter drift analyses to see if an item has become easier over time. This information helps test developers make decisions about cycling out items from production or planning new calibration studies.

Interested in learning about item analysis or how-to take your test planning to the next level? I will be presenting a series of workshops at the Questionmark Conference 2016: Shaping the Future of Assessment in Miami, April 12-15. I look forward to seeing you there! Click here to register and learn more about this important learning event. 


Questionmark Secure patent granted

Copy of patent grant imageJohn Kleeman HeadshotPosted by John Kleeman

I’m pleased to let you know that Questionmark has been granted a US patent for one of our innovations in our secure browser, Questionmark Secure.

Questionmark was one of the pioneers in secure browsers. A secure or lock-down browser is designed to help organizations provide a secure environment in which to deliver higher stakes assessments such as tests and exams. It helps prevent cheating in an assessment by disabling functions that participants could use to print or copy exam material, “accidentally” exiting a test, or gaining access to materials on their computers or the Internet that could give an unfair advantage.

Here’s a little history on how we got here:

Our first secure browser called Perception Secure Browser was produced in 1999 – you can see the press release here.

This browser, like many current secure browsers, was started up to run a specific test. But many of our customers requested something slightly different – they wanted a participant to be able to use an ordinary web browser to participate in learning courses, navigate through registration screens and/or use a learning management system, and have the secure browser launched automatically once the assessment starts. This would allow the participant to use standard browser capabilities whilst learning or registering – but when security becomes important, have the secure browser take over. Then when the assessment is over, the participant can revert to the normal browser.

Questionmark Secure splash screenThis required some clever technical work to make happen, but in 2003, we introduced  a new secure browser called Questionmark Secure which did exactly this (you can see the press release here). A participant can use a normal browser to navigate through learning or registration screens, and when they reach the secure assessment, Questionmark Secure takes over to make the assessment process secure. Our current Questionmark Secure product, though hugely improved over the 2003 version (!) uses the same concept – originally invented by Eric Shepherd, Paul Roberts and myself.

Because Questionmark Secure used innovatory technology, we filed for a patent on some of the methods and technology used and related to it . It took over 10 years to be granted but Questionmark is proud to have received U,S. Patent Number 9,055,048 recently for this unique invention.  The abstract for the patent reads:

A method for interacting with a user, comprising communicating with at least one cooperative server through a normal browser; automatically receiving encrypted data having an associated received type code indicative of a requirement for a secure browser having restricted functionality with respect to a functionality of the normal browser; selectively and automatically invoking the secure browser for handling of the received encrypted data based on the received type code associated with the received encrypted data; receiving the encrypted data with the invoked secure browser for handling thereof, wherein the received encrypted data is not available for use by the user in the normal browser and the invoked secure browser imposes restrictions on availability outside of the secure browser of decrypted data derived from the encrypted data; and communicating an input from the user, through the secure browser, to the at least one cooperative server.

For Questionmark customers, this patent helps reinforce Questionmark’s role as a leader in computerized assessment.

For more information on Questionmark Secure, see https://www.questionmark.com/content/questionmark-secure.

5 Ways to Limit the Use of Breached Assessment Content

Austin Fossey-42Posted by Austin Fossey

In an earlier post, Questionmark’s Julie Delazyn listed 11 tips to help prevent cheating. The third item on that list related to minimizing item exposure; i.e., limiting how and when people can see an item so that content will not be leaked and used for dishonest purposes.

During a co-presentation with Manny Straehle of Assessment, Education, and Research Experts at a Certification Network Group quarterly meeting, I presented a set of considerations that can affect the severity of item exposure. My message was that although item exposure may not be a problem for some assessment programs, assessment managers should consider the design, purpose, candidate population, and level of investment for their assessment when evaluating their content security requirements.

mitigating risk

If item exposure is a concern for your assessment program, there are two ways to mitigate the effects of leaked content: limiting opportunities to use the content, and identifying the breach so that it can be corrected. In this post, I will focus on ways to limit content-using opportunities:

Multiple Forms

Using different assessment forms lowers the number of participants who will see an item in delivery. Having multiple forms also lowers the probability that someone with access to a breached item will actually get to put that information to use. Many organizations achieve this by using multiple, equated forms which are systematically assigned to participants to limit joint cheating or to limit item exposure across multiple retakes. Some organizations also achieve this through the use of randomly generated forms like those in Linear-on-the-Fly Testing (LOFT) or empirically generated forms like those in Computer Adaptive Testing (CAT).

Frequent Republishing

Assessment forms are often cycled in and out of production on a set schedule. Decreasing the amount of time a form is in production will limit the impact of item exposure, but it also requires more content and staff resources to keep rotating forms.

Large Item Banks

Having a lot of items can help you make lots of assessment forms, but this is also important for limiting item exposure in LOFT or CAT. Item banks can also be rotated. For example, some assessment programs will use an item bank for particular testing windows or geographic regions and then switch them at the next administration.

Exposure Limits

If your item bank can support it, you may also want to put an exposure limit on items or assessment forms. For example, you might set up a rule where an assessment form remains in production until it has been delivered 5,000 times. After that, you may permanently retire that form or shelve it for a predetermined period and use it again later. An extreme example would be an assessment program that only delivers an item during a single testing window before retiring it. The limit will depend on your risk tolerance, the number of items you have available, and the number of participants taking the assessment. Exposure limits are especially important in CAT where some items will get delivered much more frequently than others due to the item selection algorithm.

Short Testing Windows

When participants are only allowed to take a test during a short time period, there are fewer opportunities for people to talk about or share content before the testing window closes. Short testing windows may be less convenient for your participant population, but you can take advantage of the extra downtime to spend time detecting item breaches, developing new content, and performing assessment maintenance.

In my next post, I will provide an overview of methods for identifying instances of an item breach.

Questionmark customers still safe independent of Safe Harbor

eu flagJohn Kleeman HeadshotPosted by John Kleeman

Since my earlier post, Is Safe Harbor still safe for assessment data?, the European Court of Justice has ruled that the Safe Harbor mechanism under which many transfers of personal data from Europe to the US take place is no longer valid. Here is how Questionmark customers typically remain safe in spite of this invalidation.

What is the EU-US Safe Harbor Framework?

The EU-US Safe Harbor Framework was established by the European Commission and the US government in 2000 to facilitate transfers of personal data from the EU to eligible US companies that certify to and comply with the Safe Harbor principles. You can see more about Safe Harbor at the US government website: http://www.export.gov/safeharbor/.

What did the European Court of Justice decide on 6 October 2015 regarding the EU-US Safe Harbor Framework?

Essentially, the European Court of Justice decision means that the EU-US Safe Harbor Framework does not provide a valid legal basis within the European Union for transfers of personal data from Europe to the US. The Court reached this conclusion by invalidating the European Commission’s 2000 decision approving Safe Harbor as adequately protecting personal data.

What does the European Court of Justice decision mean for the use of Questionmark OnDemand by organizations based in the EU?

Questionmark has been following these developments and has been aware of concerns about Safe Harbor for some time. Questionmark has measures in place with its non-EU subcontractors who hold OnDemand data. These arrangements include the EU Model Clauses which were not invalidated by the European Court of Justice.

If you are using our European OnDemand service, then all data is hosted in the European Union. In the rare cases that data leaves the European Union, for example for troubleshooting purposes, we have EU Model Clauses in place with any non-EU subcontractors to ensure that any such data transfer is legal, and we regularly review the security of such subcontractors.

Most EU customers of Questionmark use our European OnDemand service, but if you are an EU customer using our US OnDemand service, then this service is delivered from our US data center. However, providing your contract with or invoice from Questionmark is with Questionmark Computing Limited, the UK headquarters company of Questionmark, then you should have no cause for concern. Questionmark is legally obliged to follow UK data protection law. Also, we have EU Model Clauses in place with Questionmark Corporation, and through the corporation with the US data center that delivers the US OnDemand service. So we do not rely on Safe Harbor for personal data stored within Questionmark OnDemand.

What does the European Court of Justice decision mean for the use of Questionmark OnDemand by an organization based outside of the EU?

Organizations without EU personal data  will not be concerned about this ruling, which only applies to transfers of personal data from the EU. Questionmark continues to place the highest value on security for all our customers, and this legal ruling doesn’t change that.

If you have EU personal data and you are not based in the EU, please raise any questions you may have about this with your account manager at Questionmark. We will do everything we can to help you.

What about the US Patriot Act? Is my data stored with Questionmark vulnerable to legal action under the Patriot Act?

Unlike many technology vendors, Questionmark is headquartered in Europe. This means that the services we offer from Europe to our European customers are resistant to legal action within the US, such as under the Patriot Act.

Questionmark’s European OnDemand Service is run by a UK company using a European owned data center operator.

What if I am using Questionmark Perception?

If you are using Questionmark Perception your organization hosts the data and is responsible for compliance with local, and potentially, international laws. So so you need to seek independent legal advice as to whether your systems are configured correctly and whether your subcontractors have signed up to the EU model clauses. You will not normally need to send personal data to Questionmark, however, it may be necessary for us to ask for a copy of your Perception database to troubleshoot an issue, and if you do so, we will treat this securely. If you have any concerns about this process as a result of the Safe Harbor ruling, please raise with your account manager. You may also want to consider migrating to Questionmark OnDemand – please contact your account manager for further information.

This blog post has been written and is provided for general informational purposes only. The content of this blog does not constitute legal advice of a general or specific nature, and readers should consult an attorney to establish how these recent developments impact their organizations.

Performance testing is to certifications as simulation is to learning

Howard Eisenberg HeadshotPost by Howard Eisenberg

I just attended the Performance Testing Council Summit. Performance testing is “testing by doing.”  Exam developers create performance items that require candidates to actually perform real-world, authentic task not multiple-choice questions that have only one best answer or allow a low-ability candidate to guess the correct answer.  The outcome of the task is then evaluated to determine a score, or how well the candidate performed.

All but one of the attendees of this meeting representing certification programs were from software/IT companies. The IT domain lends itself very well to the adoption of performance testing. Advances in virtualization and software-as-a-service make it possible to provision “testing labs” with specific characteristics and traits in minutes and at low costs.  Moreover, as these labs can be hosted in the cloud nowadays, there’s no need for a candidate to travel to a specific location to take an exam.  This means that IT, performance-based certifications can and indeed ARE being delivered online and on-demand, with the help of remote proctoring and other technology-enabled security controls.

sunWe’d like to hear from you! The call for proposals is officially open for the Questionmark 2016 Users Conference. If you have an experience you would like to share with the Questionmark community, please submit a presentation proposal here: click here.

These labs are the real-world context in which an IT professional works.  It’s not a simulation of the software, tools, network connections, etc.: it is the real thing.  As such, it’s arguably a more valid methodology for assessing an IT professional’s ability to perform the tasks required by the job.  So using a performance exam for an IT certification makes sense.

Alas, not all certification exams and the professional domains they represent are as well suited to performance testing. It’s not as easy to recreate the environment in which a registered nurse performs his or her daily duties, for example.  In other domains where technology is not center-stage, Questionmark’s customers have historically done the next best thing.  What’s that, you ask?  Well, it’s simulating the performance environment within the test. And if high-fidelity simulation is not cost-effective to develop, then it’s using real-world exhibits, artifacts, and scenarios expressed through multimedia to bring as much of the performance context/environment to the test as is feasible and cost-effective.

Performance testing is to certifications as simulation is to learning.  It’s that “holy grail.”  If we can make the exam look and feel like the job, then it will have the greatest potential to be the truest measure of ability.  If we can make the training look and feel like the job, then it will have the greatest potential to adequately prepare the employee.  (I say “potential” only because the instrument or the simulation must still be well-designed).

I know that many Questionmark customers have struggled to attain this ideal. That is the reality of working with budgets, timelines and other limited resources.  But I’m willing to bet that many customers have creatively worked around these challenges to create valid tests and exams that provide solid measurement value to the programs in which they are used.

sunIf you have a story to tell about such challenges and solutions, then please share them with the Questionmark community at the Questionmark 2016 Users Conference. Click here to submit your presentation proposal. *Submission deadline is December 4. Slots are limited.

Big Shoes to Fill: A Q&A with Charlie Talmage

Julie Delazyn HeadshotPosted by Julie Delazyn

Sixteen years ago this week, Charlie Talmage started working at Questionmark. Things were a little different back then… not horse and buggy different, but Perception V 2.1 different, which is pretty crazy.

Charlie Talmage

After 16 years of working as a Questionmark technical support representative and quality assurance tester, Charlie Talmage is retiring at the end of the week.

Charlie is the kind of guy who lights up a room and makes lifelong friends, so when I heard he was retiring, or, as he put it, “[getting ready to] lay my burden down and let someone else step into my shoes, however large or small those shoes may be,” I couldn’t pass up the opportunity to talk to him about his significant experiences as a Questionmark technical support representative and quality assurance tester.

Sixteen years is a long time! How have you seen the company change since you started working here?

Good lord, the industry has come a long way, and Questionmark has been really good at keeping up and staying ahead of the curve. When I look back to what I was supporting— very straight forward and direct — a simpler product all around.  Most of our customers used Microsoft Access as their database—today the amount of information processed requires SQL or Oracle.  And Questionmark does so much more than it did back then, and the interface reflects this.  There’s so many whistles and bells that we couldn’t even imagine 16 years ago. As a tech support rep for many years, I can tell you that you have to keep learning and stay flexible. That’s what we’ve been able to do as a company. It’s a whole different ballgame from Windows ’98.

What have you enjoyed most about your time at Questionmark?

The relationships I have built with customers and co-workers over the years. There are definitely moments that stay with you. When I was a tech support rep I had a customer who I referred to as a frequent flier. He called often to discuss fixes and ask questions. Since you talk often, you develop a rapport. He was a very nice man. A St. Louis Cardinal fan. Every time we got on a call, we would spend the first three or four minutes talking baseball. One day, he called me in a huge difficult situation.  He was five minutes from delivering an exam to a large group, and things weren’t working right. I heard him out and was able to help him in jig time with two minutes to spare. That was a huge triumph. Those moments are what you live for.

 So what’s life going to be like for you after October ninth? 

Definitely a mixture of emotions. All my life I’ve wanted to travel, so I’ll be hanging out with my wife Karen and visiting Ireland, taking a train trip through the Canadian Rockies. But nature abhors a vacuum. I have a feeling I’ll wander to my office often. Questionmark has been a real nice atmosphere, with great people working together to reach a goal. It’s a family kind of deal here, and you’re all working together as a team, and in that way you get to work on some great projects. I’ve never been around a nicer group of people, and I’ve developed some great relationships.  It will be strange to shut these machines off for the last time, but I’m sure I’ll manage to find ways to fill my time.

Thank you for taking the time to speak with me Charlie. We’ll miss you around here!