Embed a Questionmark Perception assessment, survey or quiz inside your blog entries on WordPress.

• To see how this would look, see embedded quizzes within our own blog.

• Check out this How-to on our developer Web site.

• WordPress is an open source blog publishing application and can be used for basic content management. A low-stakes quiz or survey within WordPress is a great way to interact with your blog and collect information from your readers.

Please note it is only possible to include the IFrame code in a self-hosted WordPress blog.

What makes technology-enabled learning and assessment different in the rest of world? Language instantly springs to mind when we consider what sets one country apart from another. But other differences need to be considered, too, when deciding how best to use technology- enabled learning and assessments.

A recent post in Questionmark CEO Eric Shepherd’s blog explores the differences that arise when you cross social, economic and geographical boundaries.  Eric poses the question: Apart from Language, What Challenges Make Technology Enabled Learning and Assessment Different in the Rest of the World? He then identifies four key points that might drive us to use different kinds of assessments depending on where we are in the world:

1. Invalid Assumptions About Internet Connectivity (Internet connectivity will create dramatically different experiences for a student in the Amazon and a student in the USA or Europe)
2. Cost of Internet Device – The cost of purchasing a computer or PDA  in Europe or North America represents a fraction of an average annual salary, whereas in some areas of the world the cost might be 6 – 12 months of an average person’s salary. The resulting use of smaller, lower cost, generally mobile devices in poorer areas of the world calls for the re-sizing of content to accommodate them.
3. Conformance with Local Laws: Laws regarding data privacy, accessibility and equal access vary from country to country.
4. Culture: Contrasting value systems can cause different cultures to think differently about such aspects of assessment as cheating. Cheating may be thought of as solidarity within a culture that promotes collectivism and loyalty. 

For more insights on this subject and many others, visit Eric’s blog at http://blog.eric.info.

As you probably know, I live in Miami, but as I spend more than 250 nights in hotels around the world I love it when I can stay at home. This year, I am happy that the Questionmark Users Conference is in Miami March 14 – 17! So much so that I’ve posted some pictures and videos to my blog that I feel capture the spirit of Miami, South Beach and what you can expect when you attend the conference.

This year’s conference will have a strong focus on Questionmark Perception version 5 and new trends in assessment delivery.  In addition to tech training, case studies, best practice sessions and peer discussions, you’ll be able to meet one-on-one with our technicians and product managers and network with other Perception users who share your interests.

We are proud to be making this conference greener than ever. The Hilton Miami Downtown participates enthusiastically in the Florida Green Lodging Program, and we at Questionmark are doing our part for the environment by reducing the amount of paper we use, decreasing the size of our conference notebooks and making handouts available online.

I love living in Miami and think it’s a vibrant, colorful and friendly backdrop for the conference. I hope you will take the opportunity to stroll down Lincoln Road in South Beach during our Monday evening dine-around and that you’ll join us on Tuesday evening for a dinner cruise around the harbor. These events are a lot of fun, but they also have a serious purpose: helping you get to know colleagues who share your interest in assessment and create connections that will help you succeed in your work.

I can’t wait to welcome you to the conference—and I am looking forward to learning together with you. So come on down to Miami and warm up!

Early-bird registration for the conference ends today, so I hope you can sign up soon.

Posted by John Kleeman

As I take stock of the past year and look forward to the next, I’m doing so with great appreciation for all of Questionmark’s customers and users and for all my colleagues. I’d like to extend my best wishes to everyone for 2010.

I’m looking forward to the New Year as the start of a decade that promises to be an exciting one in which to work.There should be some amazing new developments during the 2010s as the Internet—which I regard as the third great invention of humankind—continues to transform assessment!

I’d suggest the first great invention was writing, allowing people to store information and knowledge and pass it on to others. Before writing, information could only be transferred by songs and ballads, or one person’s advice to another. Once writing was invented, knowledge could be written and stored, and more complex societies could start to form.

The next great invention was the printing press. Books and knowledge were not just restricted to the rich or the learned but could be communicated to all. Access to information and ideas became much easier, and this led in time to equality and freedom and our modern world.

The Internet, by bringing everyone together irrespective of geography and allowing the synergy of the crowd, rather than just from the writer to the reader, is transforming the world and society.  These are exciting and interesting times to be living and working. We don’t yet know where the Internet will take us, but it already is changing our world and bringing it together.

Working in the field of assessment, we are fortunate to be enablers for the Internet revolution. The Internet means that knowledge, learning and ideas spread faster than ever before, and we are freed from the constraints of geography. You can read this blog entry from Kansas to Karachi, at the time you want. And you can comment on it, tweet it, or ignore it and move to something better, all in just a few keystrokes.

Assessment contributes to the Internet and the Internet contributes to assessment. The Internet is about learning and assessment is the cornerstone of learning: it diagnoses what you know and need to learn, helps confirm what you have learned and helps personalize your learning path.  The promise of the Internet is that a child in the furthest corner of the world can learn from the greatest teacher, and that the potential of everyone in the world can be fulfilled. By effective use of surveys, quizzes, tests and exams, we can be part of making this happening.

The Internet also contributes to making assessment better. From online item writing workshops and item review with tools such as Questionmark Live, to delivery over the Internet and on mobile devices, and to passing and confirming assessment results online, the Internet changes every part of assessment. And this can only change and change for the better as the technology becomes more reliable and the demands for global assessment increase.

The 1990s saw the beginnings of Internet assessment (with Questionmark proud to have produced the first Internet assessment product in 1995) and the 2000s have seen Internet assessment become widespread and useful. Questionmark has an exciting announcement to make about a new version of our software in January 2010 that will make mobile assessments and assessments in frames within wikis, blogs and portals much, much easier. With news like that for starters, the 2010s look a very exciting time for Internet assessment.

Posted by Greg Pope

In my previous blog post I highlighted some of the essential things to look for in a typical Item Analysis Report. Now I will dive into the nitty-gritty of item analysis, looking at example questions and explaining how to use the Questionmark Item Analysis Report in an applied context for a State Capitals Exam.

The Questionmark Item Analysis Report first produces an overview of question performance both in terms of the difficulty of questions and in terms of the discrimination of questions (upper minus lower groups). These overview charts give you a “bird’s eye view” of how the questions composing an assessment perform. In the example below we see that we have a range of questions in terms of their difficulty (“Item Difficulty Level Histogram”), with some harder questions (the bars on the left), most average-difficulty questions (bars in the middle), and some easier questions (the bars on the right). In terms of discrimination (“Discrimination Indices Histogram”) we see that we have many questions that have high discrimination as evidenced by the bars being pushed up to the right (more questions on the assessment have higher discrimination statistics).

Overall, if I were building a typical criterion-referenced assessment with a pass score around 50% I would be quite happy with this picture. We have more questions functioning at the pass score point with a range of questions surrounding it and lots of highly discriminating questions. We do have one rogue question on the far left with a very low discrimination index, which we need to look at.

The next step is to drill down into each question to ensure that each question performs as it should. Let’s look at two questions from this assessment, one question that performs well and one question that does not perform so well.

The question below is an example of a question that performs nicely. Here are some reasons why:

• Going from left to right, first we see that the “Number of Results” is 175, which is a nice sample of participants to evaluate the psychometric performance of this question.
• Next we see thateveryone answered the question (“Number not Answered” = 0), which means there probably wasn’t a problem with people not finishing or finding the questions confusing and giving up.
• The “P Value Proportion Correct” shows us that this question is just above the pass score where 61% of participants ‘got it right.’ Nothing wrong with that: the question is neither too easy nor too hard.
• The “Item Discrimination” indicates good discrimination, with the difference between the upper and lower group in terms of the proportion selecting the correct answer of ‘Salem’ at 48%. This means that of the participants with high overall exam scores, 88% selected the correct answer versus only 40% of the participants with the lowest overall exam scores. This is a nice, expected pattern.
• The “Item Total Correlation” backs the Item Discrimination up with a strong value of 0.40. This means that of all participants who answered the questions, the pattern of high scorers getting the question right more than low scorers holds true.
• Finally we look at the Outcome information to see how the distracters perform. We find that each distracter pulled some participants, with ‘Portland’ pulling the most participants, especially from the “Lower Group.” This pattern makes sense because those with poor state capital knowledge may make the common mistake of selecting Portland as the capital of Oregon.

The psychometricians, SMEs, and test developers reviewing this question all have smiles on their faces when they see the item analysis for this item.

Next we look at that rogue question that does not perform so well in terms of discrimination-–the one we saw in the Discrimination Indices Histogram. When we look into the question we understand why it was flagged:

• Going from left to right, first we see that the “Number of Results” is 175, which is again a nice sample size: nothing wrong here.
• Next we see everyone answered the question, which is good.
• The first red flag comes from the “P Value Proportion Correct” as this question is quite difficult (only 35% of participants selected the correct answer). This is not in and of itself a bad thing so we can keep this in memory as we move on,
• The “Item Discrimination” indicates a major problem, a negative discrimination value. This means that participants with the lowest exam scores selected the correct answer more than participants with the highest exam scores. This is not the expected pattern we are looking for: Houston, this question has a problem!
• The “Item Total Correlation” backs up the Item Discrimination with a high negative value.
• To find out more about what is going on we delve into the Outcome information area to see how the distracters perform. We find that the keyed-correct answer of Nampa is not showing the expected pattern of upper minus lower proportions. We do, however, find that the distracter “Boise” is showing the expected pattern of the Upper Group (86%) selecting this response option much more than the Lower Group (15%). Wait a second…I think I know what is wrong with this one, it has been mis-keyed! Someone accidently assigned a score of 1 to Nampa rather than Boise.

No problem: the administrator pulls the data into the Results Management System (RMS), changes the keyed correct answer to Boise, and presto, we now have defensible statistics that we can work with for this question.

The psychometricians, SMEs, and test developers reviewing this question had a frown on their faces at first but those frowns were turned upside down when they realized it is just a simple mis-keyed question.

In my next blog post I would like share some observations on the relationship between Outcome Discrimination and Outcome Correlation.

Are you ready for some light relief after pondering all these statistics? Then have some fun with our own State Capitals Quiz.

Posted by Joan Phaup

The suspense is over for those of you waiting for Part 2 of our tutorial on creating and editing assessments in Questionmark Perception! This video demonstrates how to customize and control how your assessment works. Find out how to use the assessment editor to arrange time limits, security options and other settings. Learn how to establish  the look and feel of your assessment, create  jump blocks, set up feedback options, organize post-assessment email messages and perform many other tasks. 

Click here to review Part 1 of the assessment creation tutorial, and here to watch our video about creating and managing questions.

Posted by Joan Phaup

Earlier this month I shared a tutorial on creating and managing questions using Questionmark Perception. Now it’s time to move on to organizing your questions into  assessments. This video from the Questionmark Learning Cafe will show you how to create assessment folders and assign administrators to them. It will demonstrate how Perception’s assessment wizard guides you step-by-step through the process of creating surveys, quizzes, tests and exams.

What type of assessment do you want? Do you want to set a time limit for taking it?  Do you want to provide feedback to participants? How do you want to order your questions, and how many questions would like to include? Do you want to set a pass/fail threshold? The wizard will walk you through these and many other decisions as you create your assessment.

In my next post I will share another video showing how to customize your assessments and control how they work. Watch this space for Part 2!

Posted by John Kleeman

I’m reporting from the E-Assessment Live event at Loughborough University on a practical experience of crowdsourcing assessment content organized by our events team. We had a session with around 20 workstations in a room and gave everyone access via a browser to Questionmark Live, our new software-as-a-service authoring system that allows anyone with a browser to create questions easily and email them out for use in Questionmark Perception.

Most of the people in the room  were not familiar with Questionmark. We asked them all to create a question and email them to me from the system. They all logged into Questionmark Live and wrote a question on their home town which I brought into Questionmark Perception very easily, and within 20 minutes from the first question being authored we had an assessment. See below for a screenshot.

I think the availability of applications like Questionmark Live, which allow easy creation of questions by lots of people at the same time and amalgamation into an assessment, is going to make a big difference in the assessment world. Obvious ideas include getting students to create questions for each other and having SMEs brainstorm and then review questions as a group in an item writing workshop. Essentially harness the power of the crowd by letting each person contribute simultaneously rather than write items sequentially or hierarchically.

I am sure there will be ways of using crowdsourcing for questions that no one has thought of yet and this will hugely improve our productivity. Questionmark Live is free to Questionmark software support plan customers and open for anyone to evaluate. Seeing is believing, so I encourage you to try it out on our website.

Posted by Joan Phaup

Questions are the building blocks for all sorts of assessments, from surveys and quizzes to tests and exams. Whether you create items within Perception or import them from other programs, organizing them by topics helps you manage item banks effectively. Topics can be aligned with learning objectives — a big advantage in identifying knowledge gaps, measuring skills and prescribing learning and development plans.

The following feature tour  shows how to set up a topic structure and then add questions to it. It shows how to edit questions, add graphics and fine-tune the layout before previewing and saving questions. If you are not familiar with Questionmark Perception, this video will give you a basic understanding of how to create questions and put together a well-organized assessment.

You will find other simulations in Questionmark’s Learning Cafe, which includes resources about best practices as well as tutorials about the Questionmark Perception assessment management system.