How can a randomized test be fair to all?

Joan Phaup 2013 (3) Posted by Joan Phaup

James Parry, who is test development manager at the U.S Coast Guard Training Center in Yorktown, Virginia, will answer this question during a case study presentation the Questionmark Users Conference in San Antonio March 4 – 7. He’ll be co-presenting with LT Carlos Schwarzbauer, IT Lead at the USCG Force Readiness Command’s Advanced Distributed Learning Branch.

James and I spoke the other day about why tests created from randomly drawn items can be useful in some cases—but also about their potential pitfalls and some techniques for avoiding them.

When are randomly designed tests an appropriate choice?

James Parry

James Parry

There are several reasons to use randomized tests.  Randomization is appropriate when you think there’s a possibility of participants sharing the contents of their test with others who have not taken it.  Another reason would be in a computer lab style testing environment where you are testing many on the same subject at the same time with no blinders between the computers. So even if participants look at the screens next to them, chances are they won’t see the same items.

How are you using randomly designed tests?

We use randomly generated tests at all three levels of testing low-, medium- and high-stakes.  The low- and medium-stakes tests are used primarily at the schoolhouse level for knowledge- and performance-based knowledge quizzes and tests.  We are also generating randomized tests for on-site testing using tablet computers or local installed workstations.

Our most critical use is for our high-stakes enlisted advancement tests, which are administered both on paper and by computer. Participants are permitted to retake this test every 21 days if they do not achieve a passing score.  Before we were able to randomize the test there were only three parallel paper versions. Candidates knew this so some would “test sample” without studying to get an idea of every possible question. They would retake the first version, then the second, and so forth until they passed it. With randomization the word has gotten out that this is not possible anymore.

What are the pitfalls of drawing items randomly from an item bank?

The biggest pitfall is the potential for producing tests that have different levels of difficulty or that don’t present a balance of questions on all the subjects you want to cover. A completely random test can be unfair.  Suppose you produce a 50-item randomized test from an entire test item bank of 500 items.   Participant “A” might get an easy test, “B” might get a difficult test and “C” might get a test with 40 items on one topic and 10 on the rest and so on.

How do you equalize the difficulty levels of your questions?

This is a multi-step process. The item author has to make sure they develop sufficient numbers of items in each topic that will provide at least 3 to 5 items for each enabling objective.  They have to think outside the box to produce items at several cognitive levels to ensure there will be a variety of possible levels of difficulty. This is the hardest part for them because most are not trained test writers.

Once the items are developed, edited, and approved in workflow, we set up an Angoff rating session to assign a cut score for the entire bank of test items.  Based upon the Angoff score, each item is assigned a difficulty level of easy, moderate or hard and assigned a metatag to match within Questionmark.  We use a spreadsheet to calculate the number and percentage of available items at each level of difficulty in each topic. Based upon the results, the spreadsheet tells how many items to select from the database at each difficulty level and from each topic. The test is then designed to match these numbers so that each time it is administered it will be parallel, with the same level of difficulty and the same cut score.

Is there anything audience members should do to prepare for this session?

Come with an open mind and a willingness to think outside of the box.

How will your session help audience members ensure their randomized tests are fair?

I will give them the tools to use starting with a quick review of using the Angoff method to set a cut score and then discuss the inner workings of the spreadsheet that I developed to ensure each test is fair and equal.

***

See more details about the conference program here and register soon.

Third-party audits verify our platform’s security

John Kleeman HeadshotPosted by John Kleeman

2013 was a rough year for security and data privacy. The revelations of government surveillance on the Internet surprised many and revelations about credit card and other data theft have been disappointing.

How safe are your CONTENT and RESULTS?As an assessment company, Questionmark has always taken security seriously. We know that our customers care deeply about keeping assessment content and results safe, and it’s important to get third party scrutiny and review. Our own internal teams follow strict coding guidelines which include code review and security testing and internal penetration testing. But we also have third parties do testing on us, to check that we are genuinely secure – not just believing our own marketing literature!

Some of this testing is done by customers. We support customers who need to run their own penetration tests on our platform. Much of this testing we cannot share due to confidential restrictions without an NDA with a customer. But I thought it helpful to share some results from one of our third party suppliers Veracode, who provide dynamic scanning tools we use on our OnDemand platform.

We have run separate Veracode dynamic scans on Questionmark OnDemand in our US and EU data centers, and also on Questionmark Live. In all cases the score from the penetration testing has been 98 / 100 or 99 / 100.   There were two minor issues preventing a 100% score, disclosure of the operating system in use which is inevitable using the technology stack used by Questionmark and a minor cookie setting already slated for correction in our next release.

Customers who are interested in seeing more information on our Veracode scans can request this, subject to signing a suitable non-disclosure agreement (NDA). Note that Veracode is a trademark of Veracode Inc. The company does not endorse Questionmark and this blog post should not be interpreted as any validation by Veracode of Questionmark’s solutions.

tierQuestionmark continues to improve our security. Each new release includes security improvements and we are continually improving our processes. There are many layers to security, including having certified data centers, a 3 tier architecture, implementing security from start to finish in software development, and by maintaining security during release, and much more.  See my earlier article Hard and soft defences in our castle in the cloud for more on Questionmark security.

We hope that these penetration testing results will help give comfort that Questionmark takes security seriously. For more information on Questionmark security, you can see our security white paper: see here for the version relating to our European service and here for the version relating to our US service. You can also view this video about our security measures.

Learning Styles: Fact?

Doug Peterson HeadshotPosted By Doug Peterson

Are learning styles fact or fiction? There’s a lot of debate on this subject, so I’d like to join in by  presenting each side of the case: the pros in this post and the cons in my next post.

I am fascinated by the idea of learning styles, especially in the context of eLearning and instructor-led, web-based classes (where while delivered live, you don’t have physical proximity/motion/interaction between the instructor and the students).

Here are some points that favor the idea of learning styles.

This short article explains the concept nicely: Different people learn best in different ways, and effective teaching takes this into account. While as many as 71 different learning styles have been proposed, the four most common are:

  • Visual: the learner learns best by looking at things – charts, graphs, pictures, videos, etc.
  • Auditory: the learner learns best by hearing things, for example, listening to lectures or podcasts.
  • Tactile: the learner learns best by touching something.
  • Kinesthetic: the learner learns best by doing something.

Hence a visual learner, for instance, will learn better when the material is presented visually; they will not learn as well when the material is presented as a lecture.

Taking this into account as an instructor, I could design my course to accommodate all of these styles. For example, if I’m putting together an eLearning module, I would include lots of graphics and short bullet points for the visual learner. I would also include audio narration for the auditory learner. (The visual learner could turn off the audio.)

It’s a little tough to incorporate a tactile element in eLearning, but depending on the subject matter, perhaps I could have participants create an origami widget. And for the kinesthetic learner, the origami widget exercise might be useful since it is at least a little bit of movement.  At the very least, I could break my course into several very short chunks so that the kinesthetic learner could get up and move around between chunks. Or maybe I could assign a lab where they have to go to the local office supply store and research some prices–or something like that.

Wow. That’s a lot of work.

And it may not be worth it.

I’ll tell you why in my next post.

Need Customized Reports? Try Out Our OData API

Sample Attempt Distribution

Sample Attempt Distribution Reportlet

Joan Phaup 2013 (3)Posted by Joan Phaup

The standard reports and analytics that our customers use to evaluate assessment results meet a great many needs, but some occasions call for customized reports. The Questionmark OData API makes it possible to access data securely and create dynamic assessment reports using third-party business intelligence tools.

Once these reports are set up, they provide a flow of data, updating the results as new data becomes available. OData also makes it possible to cross-reference your assessment data with another data source  to get a fuller picture of what’s happening in your organization.

My recent interview with Austin Fossey goes into more detail about this, but you explore this idea yourself thanks to the OData tutorials and dashboards  on Questionmark’s Open Assessment Platform for developers.

The site provides example open source code to show how your organization could provide reportlets displaying key performance indicators from many types of assessment data.  The examples demonstrate these sample reportlets:

  • Attempt Distribution
  • PreTest PostTest
  • Score Correlation
  • Distribution

Questionmark OnDemand customers can plug in their own data to create their own reportlets, and developers can use tutorials to get detailed instructions about connecting to OData, retrieving data and creating charts.

You can also learn a lot about the power of OData at the upcoming Questionmark Users Conference in San Antonio March 4 – 7, so we hope you’ll join us there!

Ten tips on reducing test anxiety for online test-takers

Picture of lady biting her nailsJohn Kleeman HeadshotPosted by John Kleeman

I’ve been reading about test anxiety (concern or worry by participants sufficiently severe that it impacts test performance). I’d like to share some tips on how online assessment sponsors can reduce its likelihood.

When creating and delivering tests, you seek to accurately measure knowledge, skills or abilities. Often you want to check competence or mastery for workplace or compliance reasons. If some of your participants have severe test anxiety, this doesn’t just disrupt them, it makes your test less accurate in measuring real performance. You might end up failing someone who is competent, just because anxiety affects their test performance.

Many studies (for example here) report that online tests cause less test anxiety than paper ones. Here are some suggestions on reducing test anxiety:

1. Some people have anxiety about a test because they haven’t mastered the subject being tested. Provide a clear description of what each test covers before the time of the test, and provide study resources or instruction to allow people to master the subject.

2. Test anxiety can also feed on unknowns, for instance on unfamiliarity with the test or believing untrue myths. Share information about the test’s purpose and what you do to make it fair. Also share information about the content: how many questions, how the scoring works, how much time is available and so on. Explain what happens if someone fails – for instance is it possible to retake?

3. It’s hugely valuable to provide practice tests that participants can try out before the real test. This will tell them where they are strong and weak and allow them to gain confidence in a less stressful environment prior to the real test. See my article 10 reasons why practice tests help make perfect exams for other reasons why practice tests are useful.

4. Give participants an opportunity to practice using the same type of computer, mouse, keyboard and user interface as will be used for the real test. This familiarizes them with the test environment and reduces potential anxiety, particularly for those who are less computer literate. If you are using Questionmark to deliver the test, make practice sessions available with the same template settings and the same types of questions. (Sometimes this is done with a fun quiz on a different topic, just to get people accustomed to the user interface.)

5. If you provide guidance to test-takers, point to self-help resources for people who have test anxiety. ETS provide a good resource here for instance. Another resource from the University of California is here.

6. Some self-help resources suggest breathing exercises or other exercises people can follow to reduce tension for people who are anxious about tests. Provide an environment where this is practical and train your test administrators and proctors about the prevalence of test anxiety.

7. If you have a way of encouraging test takers to sleep, take exercise and eat healthily, all these things aid a rational approach to taking a test and reducing anxiety.

8. If it works in your programme, consider whether it’s worth having a series of tests rather than a single test, so there is not a single “make or break” moment for participants. A series of tests can have other benefits too. It makes cheating harder, and by spreading out learning and revision, it can make participants retain the learning better.

9. People with disabilities are more likely to suffer test anxiety. Ensure that your program of accommodations takes this into account. See this helpful article on reducing test anxiety for people with disabilities.

10. Above all, create good quality, fair tests. If you follow good practice in authoring your questions and assessments, then there is less to be anxious about, as the test will be a good measure of performance. See Questionmark’s white paper “Five Steps to Better Tests” for some helpful advice in creating tests.

Many Questionmark users provide very effective practice quizzes and tests which help reduce test anxiety, and I hope these tips are helpful, too.

I’d love to hear additional input or suggestions.

Should we formalize informal learning?

Doug Peterson HeadshotPosted By Doug Peterson

Charles Jennings a champion of the 70:20:10 learning model. He explains it in this video: 70% of learning takes place through on-the-job experience, 20% of learning is through informal relationships such as mentoring and conversations with co-workers, and only 10% of learning takes place in formal settings like classrooms and eLearning courses.

I recently read an article in Learning Solutions Magazine entitled Stop Trying to Formalize Informal Learning! The author, Stephanie Ivec, makes some good points about what I would call “organic learning” (she emphasizes unofficial, unscheduled, and impromptu) and how informal learning can possibly be negatively impacted by trying to turn it into a formal process. Ms. Ivec’s position seems to be one of “let formal be formal, and let informal be informal,” which is something with which I agree – in general.

The article got me thinking about one of my favorite sayings: “Everything in Balance.” The article made me ponder what might be the right balance between formal and informal learning.

Should we try to take every informal learning experience, codify it, and teach it in a classroom or e:Learning course? Absolutely not. As a matter of fact, it would be impossible to do so in some circumstances: There are some things you are only going to learn and get better at by doing them on the job.

So then we should just let formal be formal, and informal be informal, and never the twain shall meet, right? Well, that’s where I think we need to find balance.

Consider Henry, the sales person who has been on the job for 30 years. He’s the top sales person in the company, and for good reason: He knows the company’s products like the back of his hand, he understands the company’s customer, and he has great people skills. Henry is more than willing to work with the other members of the sales team, and he provides great advice and insights to anyone who asks. He’s even taken some of the new hires under his wing for their first couple of months to “teach them the ropes.” There is some SERIOUS informal learning going on, know what I mean?

But Henry will retire in a couple of years, and he’s taking all of that (informal) knowledge with him. Somehow, that knowledge needs to be preserved within the company. This is where a bit of formalizing would be appropriate. What if Henry were asked to write a series of blog articles on his sales techniques, what he looks for in customers, how he customizes his pitch – and then new hires were required to read those articles (and take a brief knowledge check) as part of their onboarding?

Or maybe the training department could record a series of interviews with Henry, asking the right questions and capturing the right knowledge. The results could be made available in a podcast format. This means that rather than knowledge transfer taking place randomly (a junior sales person happens to be in the break room with Henry and has the courage to ask the venerable gentleman the right question), we can take some of that informal learning and add just enough of a formal wrapper to it so that we can make sure everyone benefits. We can track the learning as well as assess it.

Another concern I have about informal learning is that we don’t always know what’s being taught. Let’s say I work in a food processing plant, and at the end of each day we go through a cleaning and sanitizing process. If I mess it up, a lot of people could end up very sick. Human nature being what it is, people tend to look for shortcuts and simple ways to do things. So proper cleaning and sanitizing training is not necessarily something that should be left to on-the-job training (70%) or learning from a co-worker (20%).

Suppose my co-worker has a shortcut that appears benign and even saves time. He believes it works just as well as the proper procedures … but it doesn’t, and it’s not something that should be propagated through the workforce and to new hires. In this situation, the company might want to use observational assessments and Job Task Analysis surveys to understand what is really happening “out on the floor.” Then Formal training could incorporate the good practices and eliminate the poor ones. An alternative might be to run high-performing employees through formal training, certify their knowledge, and then designate them as “official informal trainers” back on the job: a formal/informal training hybrid.

I think there will always be, and should always be, formal, semi-formal, and completely informal learning taking place. Training organizations should support all of these, and they can help serve the company’s goals by keeping things in balance: determining what informal training should be formalized or at least captured in a formalized wrapper as a valuable learning resource.

« Previous Page