Item Development – Five Tips for Organizing Your Drafting Process

Austin FosseyPosted by Austin Fossey

Once you’ve trained your item writers, they are ready to begin drafting items. But how should you manage this step of the item development process?

There is an enormous amount of literature about item design and item writing techniques—which we will not cover in this series—but as Cynthia Shmeiser and Catherine Welch observe in their chapter in Educational Measurement (4th ed.), there is very little guidance about the item writing process. This is surprising, given that item writing is critical to effective test development.

It may be tempting to let your item writers loose in your authoring software with a copy of the test specifications and see what comes back, but if you invest time and effort in organizing your item drafting sessions, you are likely to retain more items and better support the validity of the results.

Here are five considerations for organizing item writing sessions:

  • Assignments – Shmeiser and Welch recommend giving each item writer a specific assignment to set expectations and to ensure that you build an item bank large enough to
    meet your test specifications. If possible, distribute assignments evenly so that no single author has undue influence over an entire area of your test specifications. Set realistic goals for your authors, keeping in mind that some of their items will likely be dropped later in item reviews.
  • Instructions – In the previous post, we mentioned the benefit of a style guide for keeping item formats consistent. You may also want to give item writers instructions or templates for specific item types, especially if you are working with complex item types. (You should already have defined the types of items that can be used to measure each area of your test specifications in advance.)
  • Monitoring – Monitor item writers’ progress and spot-check their work. This is not a time to engage in full-blown item reviews, but periodic checks can help you to provide feedback and correct misconceptions. You can also check in to make sure that the item writers are abiding by security policies and formatting guidelines. In some item writing workshops, I have also asked item writers to work in pairs to help check each other’s work.
  • Communication – With some item designs, several people may be involved in building the item. One team may be in charge of developing a scoring model, another team may draft content, and a third team may add resources or additional stimuli, like images or animations. These teams need to be organized so that materials are
    handed off on time, but they also need to be able to provide iterative feedback to each other. For example, if the content team finds a loophole in the scoring model, they need to be able to alert the other teams so that it can be resolved.
  • Be Prepared – Be sure to have a backup plan in case your item writing sessions hit a snag. Know what you are going to do if an item writer does not complete an assignment or if content is compromised.

Many of the details of the item drafting process will depend on your item types, resources, schedule, authoring software, and availability of item writers. Determine what you need to accomplish, and then organize your item writing sessions as much as possible so that you meet your goals.

In my next post, I will discuss the benefits of conducting an initial editorial review of the draft items before they are sent to review committees.

Integrating and Connectors – SharePoint

Doug Peterson HeadshotPosted By Doug Peterson

There’s not just one way to integrate Questionmark with your SharePoint portal. There’s not just two ways. There are actually three ways to integrate a Questionmark assessment into a SharePoint page!

For Perception (on-premise) customers, it’s possible to use Windows Authentication to present to a SharePoint user a list of assessments for which they have been scheduled – without having to re-authenticate the user in Questionmark.

Questionmark has also developed a SharePoint Connector for our OnDemand customers. It’s a SharePoint web part that automatically logs the user into Questionmark and displays a list of assessments for which they have been scheduled.

The third way to integrate a Questionmark assessment with a SharePoint page is to embed it in the page. This is great for simple, anonymous quizzes and knowledge checks.

Check out this video for a quick overview of all three methods of integrating Questionmark and SharePoint.

SharePoint Video

Integrating and Connectors – Blackboard

Doug Peterson HeadshotPosted By Doug Peterson

So far in this series we have discussed integrating using common standards – launch-and-track with AICC or SCORM, or a tighter integration with the Questionmark LTI Connector. In this installment we take a look at a deeper, custom integration – the Questionmark Blackboard Connector.

The latest version of Blackboard does have LTI capabilities, but we recommend using our Blackboard Connector instead of the LTI Connector as the Blackboard Connector has more functionality. As you’ll see in the following video, the Blackboard Connector handles a number of things behind the scenes – automatically creating groups that represent courses, adding participants and instructors to the appropriate groups, scheduling, etc. You also have a great amount of control – the Blackboard Connector has settings that allow you to control which courses and/or participants can interact with Questionmark from the Questionmark side, instead of automatically synchronizing everything.

Enjoy this video about integrating Questionmark with Blackboard using the Questionmark Blackboard Connector, and let me know if you have any questions!

Integrating Blackboard

Should we formalize informal learning?

Doug Peterson HeadshotPosted By Doug Peterson

Charles Jennings a champion of the 70:20:10 learning model. He explains it in this video: 70% of learning takes place through on-the-job experience, 20% of learning is through informal relationships such as mentoring and conversations with co-workers, and only 10% of learning takes place in formal settings like classrooms and eLearning courses.

I recently read an article in Learning Solutions Magazine entitled Stop Trying to Formalize Informal Learning! The author, Stephanie Ivec, makes some good points about what I would call “organic learning” (she emphasizes unofficial, unscheduled, and impromptu) and how informal learning can possibly be negatively impacted by trying to turn it into a formal process. Ms. Ivec’s position seems to be one of “let formal be formal, and let informal be informal,” which is something with which I agree – in general.

The article got me thinking about one of my favorite sayings: “Everything in Balance.” The article made me ponder what might be the right balance between formal and informal learning.

Should we try to take every informal learning experience, codify it, and teach it in a classroom or e:Learning course? Absolutely not. As a matter of fact, it would be impossible to do so in some circumstances: There are some things you are only going to learn and get better at by doing them on the job.

So then we should just let formal be formal, and informal be informal, and never the twain shall meet, right? Well, that’s where I think we need to find balance.

Consider Henry, the sales person who has been on the job for 30 years. He’s the top sales person in the company, and for good reason: He knows the company’s products like the back of his hand, he understands the company’s customer, and he has great people skills. Henry is more than willing to work with the other members of the sales team, and he provides great advice and insights to anyone who asks. He’s even taken some of the new hires under his wing for their first couple of months to “teach them the ropes.” There is some SERIOUS informal learning going on, know what I mean?

But Henry will retire in a couple of years, and he’s taking all of that (informal) knowledge with him. Somehow, that knowledge needs to be preserved within the company. This is where a bit of formalizing would be appropriate. What if Henry were asked to write a series of blog articles on his sales techniques, what he looks for in customers, how he customizes his pitch – and then new hires were required to read those articles (and take a brief knowledge check) as part of their onboarding?

Or maybe the training department could record a series of interviews with Henry, asking the right questions and capturing the right knowledge. The results could be made available in a podcast format. This means that rather than knowledge transfer taking place randomly (a junior sales person happens to be in the break room with Henry and has the courage to ask the venerable gentleman the right question), we can take some of that informal learning and add just enough of a formal wrapper to it so that we can make sure everyone benefits. We can track the learning as well as assess it.

Another concern I have about informal learning is that we don’t always know what’s being taught. Let’s say I work in a food processing plant, and at the end of each day we go through a cleaning and sanitizing process. If I mess it up, a lot of people could end up very sick. Human nature being what it is, people tend to look for shortcuts and simple ways to do things. So proper cleaning and sanitizing training is not necessarily something that should be left to on-the-job training (70%) or learning from a co-worker (20%).

Suppose my co-worker has a shortcut that appears benign and even saves time. He believes it works just as well as the proper procedures … but it doesn’t, and it’s not something that should be propagated through the workforce and to new hires. In this situation, the company might want to use observational assessments and Job Task Analysis surveys to understand what is really happening “out on the floor.” Then Formal training could incorporate the good practices and eliminate the poor ones. An alternative might be to run high-performing employees through formal training, certify their knowledge, and then designate them as “official informal trainers” back on the job: a formal/informal training hybrid.

I think there will always be, and should always be, formal, semi-formal, and completely informal learning taking place. Training organizations should support all of these, and they can help serve the company’s goals by keeping things in balance: determining what informal training should be formalized or at least captured in a formalized wrapper as a valuable learning resource.

Integrating and Connectors – Moodle

Doug Peterson HeadshotPosted By Doug Peterson

This installment of the Integrating and Connectors series focuses on Moodle. Technically, it’s really about the Questionmark LTI Connector and how it can be used to integrate with Moodle. (We’ll take a look at integrating with Canvas using the LTI Connector in a future installment.)

LTI stands for Learning Tools Interoperability. LTI is a specification published by the IMS Global Learning Consortium with the goal of providing a way for different learning tools to talk to each other and work together. Moodle (a Learning Management System, or LMS) and Questionmark (an Assessment Management System, or AMS) integrating their functionality is a perfect example of the concept.

So far in this series, we’ve looked at using SCORM or AICC to do a simple launch-and-track, and in the case of SuccessFactors, a simple Single Sign On (SSO) from the Learning LMS into the Questionmark Enterprise Manager. This is a very high-level integration. The assessment is simply launched and reports back to the source of the launch. The SuccessFactors SSO requires some manual intervention to set up an admin ID within Questionmark – the connection doesn’t just happen “automagically”. The LTI Connector allows for a much deeper integration.

As you’ll see in this video, once the LTI Connector is configured in the Moodle environment, a Moodle instructor can log into Moodle and add a Questionmark assessment to a course – from within Moodle, without having to have an ID and password and log into Questionmark at all.

Similarly, a student can log into Moodle and launch a Questionmark assessment – again, from within Moodle, without a second set of credentials. Furthermore, an instructor can also use Questionmark’s authoring and reporting functionality – you guessed it – all from within Moodle.

The LTI Connector allows for a deep integration with Moodle, giving the instructor and student a seamless experience in what behaves to them like a single environment, even though they are actually moving back and forth between Moodle and Questionmark.

qm and moodle vid

Six trends to watch in online assessment in 2014

John Kleeman HeadshotPosted by John Kleeman

As we gear up for 2014, here are six trends I suggest could be important in the coming year.

1. Privacy. The revelations in 2013 that government agencies intercept so much electronic data will reverberate in 2014. Expect a lot more questions from stakeholders about where their results are stored and how integrity, data protection and privacy are assured, including the location and ownership of suppliers and data centres. I suspect some organizations will look to build trust with stakeholders by adopting the ISO standard on assessments in the workplace ISO 10667.

2. Anticipation of problems. Many organizations already use assessments to look forward, not just backwards. In regulatory compliance, smart organizations don’t just use assessments to check competence; they analyze results from assessments to identify trends or problems that can indicate potential issues or weaknesses, and prompt corrective measures before it gets too late. Universities and colleges increasingly use assessments to predict problems and help prevent students from dropping out (see for instance Use a survey with feedback to aid student retention). It’s exciting that assessments can be used to find issues in this way and deal with them before they happen. Don’t just treat assessments as a rear-view mirror: use them to look forward.

3. Software as a service (SaaS). For all but the very large organizations, running online assessments via a software as a service is much more cost-effective than running an on-premise system. Delegating to a service provider like Questionmark,makes the hassle of upgrading, maintaining security patches and managing deployment goes away. Increasingly, delivering assessments via a SaaS model will become the default.connected

4. Smaller and more connected world. The Internet is bringing us all together. The world is becoming connected, and in some sense smaller. We can no longer think of another continent or country as being a world away, because we can all connect together so easily. This means it is increasingly important to make your assessments translatable, multi-lingual and cross-cultural. Most medium and large organizations work across much of the world, and assessments need to reflect that.

5. Environment. I wonder if 2014 could be the year when the environmental benefits of online assessments could start to be seriously recognized. Clearly, using computers rather than paper to deliver assessments saves trees, but a bigger benefit is in reduced carbon emissions due to less traveling. For service organizations, business travel is a large proportion of carbon emissions (see for example here), and delivering training and assessments online can make a useful difference. With many countries requiring reporting of carbon emissions by listed companies, this could be important.

6. Security. Last but definitely not least, assessment security will continue to matter. As there is more awareness of the risks, everyone will expect high levels of technical and organizational security in their assessment delivery.  If you are a provider, expect a lot more questions on security from informed users; and if you are a customer or user, check that your supplier and your internal team is genuinely up to date on its security.

Read this list and look at the starting letters, and you get P – A – S – S – E – S! I wish you a happy new year and hope that each of your test-takers passes their assessments in 2014 when it is appropriate that they do so.

« Previous PageNext Page »