Can online quizzes before lectures increase reading by literature students?

John Kleeman HeadshotPosted by John Kleeman

It’s often suggested in higher education circles that computer-assisted assessment is more useful in scientific subjects than in the humanities.

I’d like to share a counter view from some research by Dr Judith Seaboyer at the University of Queensland. She presented a paper at the 2013 International Computer Assisted Assessment conference about how computerized quizzes can help greatly in teaching English Literature.

One challenge of Literature courses is ensuring students read required texts in advance of lectures: Sometimes students struggle to make time for necessary reading, but if they fail to do it, they will likely struggle later on in essays and exams.

Dr. Seaboyer’s solution? Require students to take an online quiz before each lecture. Students must complete the quiz before midnight the night before the first lecture on a text. The quiz, which includes 6 questions chosen at random from a pool of about 15, gives a small amount of course credit. The questions are as Google and eBook search-proof as possible: using different words to those in the text, so they require real reading and understanding.

Here, for example, is a question about Ian McEwan’s Atonement:

Where does Robbie notice a human limb, the memory of which will return to haunt him?

(a) in the fork of a tree

(b) in a Joe Lyons tea house

(c) on the beach at Dunkirk

The right answer is (a), but this would not be easy to identify by searching, as the book mentions a human “leg” not a “limb” and the other answers are plausible. Unless you’ve read the book recently, you will struggle to answer.

Students reported that the online quizzes motivated them to complete assigned reading before the lecture as can be seen in the survey result below:

"The online quiz motivates me to complete assigned reading before the lecture" Mean 4.31 on Strongly Agree to Strongly Disagree Likert Scale

Dr. Seaboyer’s  preliminary research suggests that 83% of first year English students read at least 5 out of 6 books in a course where quizzes were used as against around 45% in a control group.

I’ve seen other examples of quizzes encouraging learners to access learning material that they might otherwise put off until later, and I’d encourage others to consider this approach.

To quote Dr. Seaboyer:

“Computer-assisted assessment can result in more reading and persistent, careful, observant, resilient reading that leads to critical engagement.”

She also believes that this could also apply across a range of other disciplines as well as Literature.

What will you take to San Antonio?

Joan Phaup HeadshotPosted by Joan Phaup

river walk

Riverwalk, San Antonio

We’ve had an enthusiastic response to our announcement of San Antonio as the location for the Questionmark 2014 Users Conference March 4 – 7. We’ll be at the Grand Hyatt, right next to the famous Riverwalk and within easy walking distance of many local attractions.

But the most important thing on our minds is the conference program. Results of a recent customer survey will help us provide content that addresses current concerns and questions about assessment-related best practices, as well as the use of Questionmark technologies and services.

Equally significant will be the content created by Questionmark users themselves — people who present case studies or lead discussions.

We are  seeking case study and discussion proposals from now until October 18, so consider what you’d like to contribute. Please note that presenters will receive some red carpet treatment — including a special dinner in their honor on Tuesday, March 4th. And we award one 50% registration for each case study presentation.

Now is a good time to ask yourself what you could bring to the conference:

doug teaching 2013

2013 Users Conference

  • An account of how you are using assessments to support organizational goals?
  • Some lessons learned and advice for others?
  • A unique application of online or mobile assessments?
  • An account of how you have integrated Questionmark with another system?
  • A topic you think would be worth discussing with colleagues?

Click here for more details and proposal forms. Even if you are not sure you’ll attend the conference, we would like to hear from you! And whether you plan to present or not, plan now to have the conference in your budget for 2014. You will find information about conference return on investment and an ROI tookit here.

Test Security at Shenandoah University: A SlideShare Presentation

Headshot JuliePosted by Julie Delazyn

Students involved in Shenandoah University’s mobile learning initiative, iMLearning, use the Apple MacBook Pro for accessing course material and online assessments. They can also choose an iPodTouch, iPad 3G or iPhone for a quick mobile delivery option. Using the iPad for lab exercises, for instance, students can move from station to station without having to carry around a laptop.

In this presentation from the 2013 Questionmark Users Conference, Terra Walker and Cheri Lambert of Shenandoah University explore various security measures they use to ensure test integrity both in the iMLearning program and Windows-based testing. The presentation focuses on Using Questionmark Secure with Windows PCs and Macs but also covers measures such as question and answer randomization, seating arrangements, proctors and honor code.

This is just one example of what people learn about at our Users Conferences. Registration is already open for the 2014 Users Conference March 4 – 7 in San Antonio, Texas. Plan to be there!

mLearning’s about content – not devices, platforms or tools

Jim Farrell HeadshotPosted by Jim Farrell

mLearnCon is a relatively new conference (about 3 years old) produced by the eLearning Guild. I attended it recently in San Jose, California, and had a great time there. If you are wondering if this conference would benefit your organization, consider these questions from the Guild’s website:

  1. Is mLearning right for our organization?
  2. What tools are best for developing engaging learning?
  3. Should we build a platform-specific application or a mobile optimized website?

While the importance of these questions may be case-by-case specific, I’ll share my thoughts about each of them in a moment. The one thing I took away from this conference is that the most important thing is not the device, platform or development tool. It is the content. I think our friend Jason Haag (@mobilejson) from the ADL said it best: “We need to start thinking about being “learning designers” and not just instructional designers, because we now have an opportunity to design for more than just formal courses in the cognitive domain. “ So yes, indeed, it’s all about content!

Here are my own responses to the Guild’s questions:

1. Is mLearning right for your organization?

I think there are two key questions behind this one: Are people in your organization using mobile to access internal content? Are people in your organization taking assessments or filling our surveys on mobile devices? (If you don’t know how to respond, adding Google Analytics can help you figure this out quickly.) Given the fact that mobile devices are everywhere these days, your answer to both of my questions is probably yes. But you need to understand WHY people are accessing your content via mobile, then develop your content in digestible chunks or in ways that solve business problems. Learning via mobile is most often pull learning done at a moment of need to support performance. So, yes! It is the content that matters.

2. What tools are best for developing engaging learning?

I love the word “engaging” in this question, but I think it is the wrong word. I would replace it with efficient, productive or correct. As a manager I am not worried about being engaging, I am worried that people do their jobs correctly and efficiently. There are lots and lots of rapid development tools out there, and mLearnCon was ripe with them. There is no be-all end-all tool for developing learning. Never was, never will be. You have to find the tools that best fit each situation and medium you are producing for. Ask the question, “How will people be consuming my content?” — then pick your tool accordingly.

3. Should we build a platform specific application or a mobile optimized website?

It seems to me this question has already been answered. Responsive design allows people visiting your content to get appropriate views. Apps are not the answer. Are you ready to support all of the growing list of mobile operating systems?

Jason Haag shared a terrific quote during a presentation he gave at the conference: “Not every mobile device will have your app on it, but every mobile device will have a browser.” Although HTML5 is not an official standard until the end of 2014, there are techniques available to detect the features for the browser and display the best possible presentation to the user. At Questionmark we like to say Author once, Schedule once, and Deliver to any device. That is truly due to using responsive design to give participants the best experience possible.

The great thing about conferences is about meeting people you admire or follow virtually via Twitter. This conference was no different. I got the chance to meet the famous Sarah Gilbert @melsgilbert. Read her cover article this month in Training and Development magazine if you are truly a beginner in the mobile learning world.

OData Tutorials with Excel PowerPivot

Austin FosseyPosted by Austin Fossey

In case you missed this earlier post, Questionmark has an OData feed that allows users to get direct access to data in their Results Warehouse—the same data source that is used to drive the reports in Questionmark Analytics.

If you have not done so, I encourage you to check out our OData API for Analytics video to learn how to connect to the OData feed, import data and create a simple report.

There are three major benefits of having this access to the Results Warehouse data:

  1. You can grab your data anytime you wish to explore a new research question about your assessment results
  2. You can manipulate those data and run your own analyses or transformations in any way you see fit
  3. You can feed those data into your own reporting tools or many other programs that already consume OData

Of course, the only reason we create assessments and collect data is so that we can make informed decisions, and the data in the OData feed just come across in their raw state.

To help you make sense of the data, we are developing short video tutorials that walk through introductory OData examples using the free PowerPivot add on for Excel.

These tutorials will cover the following topics:

  • Creating a response matrix with OData
  • Using OData to analyze the distribution of correct choices in the items on an assessment
  • Calculating a new variable in PowerPivot using the data from the OData feed
  • Using OData to generate a list of scores for participants who took multiple forms of an assessment

We will announce these new videos in the blog in the coming weeks, and you will be able to find them in the Questionmark Learning Café.

Odata 1

If you have a research question that you think could be addressed with OData, please let us know! Your suggestion may lead to additional tutorials that will help other people expand their own research.

Keep pace with assessment news and trends at European learning events

Chloe MendoncaPosted by Chloe Mendonca

bb2We recently held a Breakfast Briefing at London Microsoft, where Questionmark users and other assessment and measurement professionals  discussed the various ways in which online assessment helps to ensure regulatory compliance.

The briefing included some stimulating questions and answers and a demonstration of our browser-based authoring tool, Questionmark Live. Stephen Kelly from The London Fire Brigade, shared how England’s largest fire service uses Questionmark for training confirmation and high-stakes exams.

If you missed the meeting, you may view the presentation slides here.

bb1Events like this one offer an opportunity to see our technologies in action, speak with Questionmark experts, and meet with  individuals from a wide range of industries.

The 2013 Questionmark European Users Conference is another great learning opportunity. It packs two days of learning, networking and fun into one memorable occasion. This year the conference will take place November 10-12 in Barcelona.

Whether you’re new to Questionmark or have been using our technologies for years, there are plenty of reasons to attend this event! And if you sign up by July 15th you will save €130. Click here to register now.