Posted by Steve Lay

Last month I wrote about our new Community Edition program under which we’ll be releasing selected integration products as free-to-download Community Editions distributed under open source licenses.

Although this program represents an exciting new way to involve the development community, Questionmark is no stranger to working with open source. Moodle is a popular learning management system that is developed by an active community of open source developers. Questionmark has always distributed its connector products for Moodle under a similar open source license, so it is no surprise that our new connector for Moodle 1.9 is available as a Community Edition.

The connector is written in the widely-used PHP system used by Moodle itself and uses QMWISe (Questionmark Web Integrated Services Environment) and PIP (Perception Integration Protocol) to allow Perception tests to be included as activities directly within Moodle courses.

You can find our more information about the connector and the development project behind the Community Edition from our new developer site: developer.questionmark.com. On the site, you’ll find information about our Application Programming Interfaces (APIs) and all of our Community Edition projects. The Community Edition Moodle Connector page on the development site also links to the development project, which is hosted on the popular SourceForge system where you can download the latest version and even browse the code!

And finally, we’ve used Moodle as a development platform for the developer site itself, which means that we can demonstrate the connector working directly from the site. Look for the “Try It Out!” section on the Community Edition Moodle Connector page and see if you can answer my quick quiz on the main features of our Community Edition release program.

The hits keep coming. This month we have improved the look and feel of Questionmark Live with new buttons and improved Question Set sharing. Now when you create one shared question set, you can create another with the same shared permissions. You can also invite someone to join Questionmark Live via a shared question set or stop sharing a particular question set.

These new features are sure to make it easier for you to work collaboratively with your Subject Matter Experts (SMEs), and they’ll help you set up multiple question sets (topics) for item writing workshops. 

Watch the short video below to see how easy it is to create a new question set with sharing already set up!

Last week I wrote about tips for protecting  intellectual property.  It’s equally important keep people from cheating on tests, so here are three pointers on that subject.  I’ll be following this up with more tips in future posts.  Leave me your comments; we can always add to lists like these!

.Screening tests

Consider givinig a small pre-screening test to prevent people from taking an assessment that is beyond their current ability level. If a participant can‘t answer a certain number of these questions correctly they will not be allowed to see the remainder of the assessment. When the time does come for them to take the test,  they will not have already seen its content.

Candidate agreements

Candidate agreements or examination honor codes require  a participant’s agreement before they start an assessment, say by clicking on an “OK “ or  “Yes” button after reading the exam’s code of conduct.

The code might say something like this:  “I agree to answer the questions on this assessment without obtaining assistance from another person or via electronic means. I agree to not to share my answers with anyone during or after the exam. I further agree to not memorize or otherwise steal the intellectual property contained in this exam. I accept that if any of these conditions are violated, my exam results will be set to a zero, I will not be able to retake the exam for a period of 10 years, and I may be charged with a crime under regional laws.”

Here are some topics you might want to cover in a candidate agreement:

• The test vendor will have the option to terminate the assessment if suspicious behavior is detected
• The candidate must abide by the rules of the test center, organization, or program
• The candidate will not provide false ID or false papers
• The candidate cannot take the test on behalf of someone else
• The candidate will not engage in cheating in any form
• The candidate will not help others cheat by disclosing information about the assessment
• The candidate will not use aids that are not allowed
• The candidate will not solicit someone else to take the test
• The candidate will not cause a disturbance in the testing center
• The candidate will not tamper with the test center in any way
• The candidate will not share information

Limiting content exposure/leakage

In order to limit the amount of question content being shown to a participant at any given time, think about using question-by-question templates. These present questions one at a time to participants so that exam content is not completely exposed on screen. Participants who may intend to take pictures of the exam content or otherwise steal intellectual property will not be able to do so all at once.

There are many fine resources for learning how to prevent cheating. Two of thes are books by Dr. Gregory Cizek:  Cheating on Tests: How to Do It, Detect It and Prevent It and Detecting and Preventing Classroom Cheating: Promoting Integrity in Assessment.

There’s also our white paper: “Delivering  Assessments Safely and Securely,” and of course this blog! Watch for more security tips in my future posts.

Protecting the intellectual property contained in a test or exam is essential, not only because of the time, effort and cost of creating assessments but also because IP theft undermines the accurate measurement of knowledge and skills.

Protecting intellectual property protects the credibility of tests. Here are four tips for helping to ensure the security of intellectual property:

Create and administer multiple test forms

Rather than having only one form of the assessment being administered, delivering multiple forms of the same exam can help limit item exposure. This method also allows for the possibility of interspersing large-scale integrated beta test questions within the forms to collect psychometric information on newly developed questions.

Restrict and control administration of beta test items

Beta testing questions is an important part of high-stakes assessment, ensuring the psychometric quality of questions before they appear on actual assessments. However, it is vital that a well conceptualized beta test model is in effect to limit the exposure of newly developed questions to participants.

Update exam forms periodically

Letting exam forms become stale can over-expose questions to participants, increasing the likelihood of IP theft. An organization could consider retiring old exam forms and turning them into exam prep materials that can be sold to participants. In this way, participants could periodically expect new practice questions.

Produce exam prep materials

Organizations should consider making exam prep materials available to participants before an assessment. This will help reduce the demand for participants to try to obtain exam questions via illegal means as they will have access to the type of questions that will be asked on the actual assessment.

For more details on this subject, plust information about various means for deploying a wide range of assessment types with assurance, download our White Paper: Delivering Assessments Safely and Securely.

Just over six months ago we announced Questionmark Live at the Questionmark Users Conference in Memphis.Since then, there have been 25,000 questions created using this new browser-based tool.  That sounds great, doesn’t it! But I was surprised to meet a number of customers at our Breakfast Briefings and User Group Meetings who had not yet seen this easy way to author questions. So I’d like to turn back the clock and re-announce the availability of Questionmark Live, which  all of our Software Support Plan customers can use free of charge.

Anyone you want questions from can have unlimited access to this tool. There’s nothing to download. They can just start creating questions sets on a desired theme  or topic, using seven different question formats. The questions can utilize multimedia, links and choice-based feedback.

Do you utilize workflows that involve SMEs, instructional designers and editors? Questionmark Live offers a great way to help people work together. Users can share their question sets with each other for true collaboration. Every question created and edited has a full revision history that can be used to compare revisions and roll back to previous versions.

If you haven’t taken a look at this exciting and dynamic tool, just click here to get started. If you would like a demonstration, please email customercare@questionmark.com to set one up!

Assessments provide a valuable tool for helping organizations properly design effective and useful learning experiences. Doing so involves a five-step process.

Step 1: Define the objectives. An objective might be to increase customer satisfaction, reduce error rates or improve safety.

Step 2: Ask what knowledge and skills are required to meet the objectives. In a college or university course on organic chemistry, for example, it’s important to ask what are the knowledge and skills required to comprehend those concepts and use them. The answers to that question will help the professor establish the topic structure to define the knowledge, skills, abilities and attitudes required to meet the objectives.

Step 3: Run a needs analysis survey or skills gap survey. Here, people take a needs assessment to reveal the knowledge and skills they already have as well as what they still need. A gap analysis can be derived from the difference between what is required and people’s current knowledge and skills.

Step 4: Develop a learning plan. That plan will describe the learning objectives and explain how the plan will be administered. The learning objectives will guide the production of learning materials and assessments. Facilitating the learning process might involve instructor-led training, coaching by managers, or e-learning courses.

Step 5: Conduct a pre-learning assessment. The pre-learning assessment will have two purposes: to create intrigue about the course and  to guide each participant to the right learning experience. Advanced and novice learners will require different approaches.

To learn more about  how instructors and organizations can use assessments to improve learning, download the white paper Assessments through the Learning Process.

Posted by Greg Pope

I had a great time putting together an eight-part series on Item Analysis Analytics for this blog and was pleased with the interest it received.

When a reader asked if it would be possible to present all the posts in a single document I thought hey, let’s present the content of these articles in the form of a Questionmark White Paper! So here it is for you to download with our compliments.

I hope the paper helps you in your efforts to create test questions that make the grade!

In talking with some of our customers last week, I was reminded how valuable it can be to offer participants topic based feedback.

Obviously, everyone wants to know whether they’ve passed or failed a test. And most people look at their feedback on questions they got wrong, to understand how to improve. But you can get a single question wrong for many different reasons including misunderstanding the question, making a mistake or having a tiny gap in knowledge. However, if you score weakly in a topic area, it very likely means you have a weakness in that area that needs addressing.

In many ways, topic feedback is the Cinderella in the feedback world. Everyone expects assessment level feedback and item level feedback (perhaps it’s unfair to call them ugly sisters because they are useful and valuable), but there is a huge and often untapped learning value in topic feedback. For pre tests, post course tests, quizzes during learning and practice tests particularly, topic feedback is vital.

Suppose someone is taking an assessment in health and safety and they score 66% and this is a passing score. Sounds good! But what happens if, as in the screenshot below, they’ve scored very well in some topics and poorly in others?

In this example, the fact that someone is very weak in electrical safety could well be concerning. (Don’t let them set up the lighting for the Holidays Ball!)

It’s obvious that you want to give people feedback on the topic level, but in many tools this isn’t as easy to set up as it should be. Questionmark Perception can be a fairy godmother for topic based feedback. There are lots of easy-to-use capabilities to present topic based feedback. Here are some links to support resources to help you create topic feedback in Perception.

• You can easily create topic outcomes with feedback for different topic scores in Authoring Manager
• You set these as standard at the topic level, but can override to adjust on a per assessment level
• You can also make a topic a prerequisite for passing an assessment (for instance to prevent someone passing an assessment unless they get 60% or a specified score in key topics).
• If you want to display only some topics in the list and not all, for instance if some of your topics aren’t meaningful to the participant, you can define which topics are reported on.
• And then the feedback is displayed to participants at the end of assessment easily as in the screenshot above.

I hope you find this useful in getting your topic feedback working and helping your learners achieve their full potential.

Is it possible to build the perfect assessment design? Not likely, given the intricacies of the learning process! But a white paper available on the Questionmark Web site helps test authors respond effectively to the inevitable tradeoffs in order to create better assessments.

Measuring Learning Results, by Dr. Will Thalheimer of Work-Learning Research, considers findings from fundamental learning research and how they relate to assessment. The paper explores how to create assessments that measure how well learning interventions are preparing learners to retrieve information in future situations—which as Will states it is the ultimate goal of training and education.

The eight bits of wisdom that conclude the paper give plenty of food for thought for test designers. You can download the paper to find out how Will arrived at them.

1. Figure out what learning outcomes you really care about. Measure them. Prioritize the importance of the learning outcomes you are targeting. Use more of your assessment time on high-priority information.

2. Figure out what retrieval situations you are preparing your learners for. Create assessment items that mirror or simulate those retrieval situations.

3. Consider using delayed assessments a week or month (or more) after the original learning ends—in addition to end-of-learning assessments.

4. Consider using delayed assessments instead of end-of-learning assessments, but be aware that there are significant tradeoffs in using this approach.

5. Utilize authentic questions, decisions, or demonstrations of skill that require learners to retrieve information from memory in a way that is similar to how they’ll have to retrieve it in the retrieval situations for which you are preparing them. Simulation-like questions that provide realistic decisions set in real-world contexts are ideal.

6. Cover a significant portion of the most important learning points you want your learners to understand or be able to utilize. This will require you to create a list of the objectives that will be targeted by the instruction.

7. Avoid factors that will bias your assessments. Or, if you can’t avoid them, make sure you understand them, mitigate them as much as possible, and report their influence. Beware of the biasing effects of end-of-learning assessments, pretests, assessments given in the learning context, and assessment items that are focused on low-level information.

8. Follow all the general rules about how to create assessment items. For example, write clearly, use only plausible alternatives (for multiple-choice questions), pilot-test your assessment items to improve them, and utilize psychometric techniques where applicable.