Q&A: Pre-hire, new-hire and ongoing assessments at Canon

HollyGroder

Holly Groder

Headshot JuliePosted by Julie Delazyn

Holly Groder and Mark Antonucci are training developers for Canon Information Technology Services, Inc. (Canon ITS). During their case study presentation at the Questionmark 2015 Users Conference in Napa Valley March 10-13, they will talk about Leveraging Questionmark’s Reports and Analytics Tools for Deeper Insight.

Their session will explore Canon’s use of assessments in hiring, training, continuing job skills assessment and company-wide information gathering via surveys.

I asked them recently about their case study:

Why did you start using Questionmark? 

The primary reason for seeking a new assessment tool was our desire to collect more information from our assessments, quicker. Questionmark offered the flexibility of web-based question creation and built-in reports. Questionmark also offered the ability to add jump blocks and a variety of templates. The survey capabilities were just a bonus for us. We were able to streamline our survey process to one point of contact and eliminate an additional software program.

What kinds of assessments do you use?

MarkAntonucci1

Mark Antonucci

The principal function is split between four business needs: Pre-hire employment assessments, new-hire or cross-training assessments, continuing job knowledge assessments, and business information gathering (surveys).

How are you using those tools?

First, potential employees are required to participate in a technical knowledge assessment prior to an offer of employment. Once employment has been offered and accepted, the new employees are assessed throughout the new-hire training period. Annually, all call center agents participate in a job skills assessment unique to their department. And finally, all employees participate in various surveys ranging from interest in community events to feedback on peer performance.

What are you looking forward to at the conference?

We are interested in best practices, insight into psychometrics, and, most important, networking with other users.

Thank you Holly and Mark for taking time out of your busy schedules to discuss your session with us!

***

If you have not already done so, you still have a chance to attend this important learning event. Click here to register.

Evidence that assessments improve learning outcomes

John Kleeman HeadshotPosted by John Kleeman

I’ve written about this research before, but it’s a very compelling example and I think it’s useful as evidence that giving low stakes quizzes during a course correlates strongly with improved learning outcomes.

The study was conducted by two economics lecturers, Dr Simon Angus and Judith Watson, and is titled Does regular online testing enhance student learning in the numerical sciences? Robust evidence from a large data set. It was published in the British Journal of Educational Technology Vol 40 No 2, 255-272 in 2009.

Angus and Watson introduced a series of 4 online, formative quizzes into a business mathematics course, and wanted to determine whether students who took the quizzes learned more and did better on the final exam than those who didn’t. The interesting thing about the study is that they used a statistical technique which allowed them to estimate the effect of several different factors, and isolate the effects of taking the quizzes from the previous mathematical experience of the students, their gender and their general level of effort to determine which impacted the final exam score most.

You can see a summary of their findings in the graph below, which shows the estimated coefficients for four of the main factors, all of which had a statistical significance of p < 0.01.

Factors associated with final exam score graph

You can see from this graph that the biggest factor associated with final exam success was how well students had done in the midterm exam, i.e. how well they were doing in the course generally. But students who took the 4 online quizzes learned from them and did significantly better. The impact of taking or not taking the quizzes was broadly the same as the impact of their prior maths education: i.e. quite reasonable and significant.

We know intuitively that formative quizzes help learning, but it’s nice to see a statistical proof that – to quote the authors – “exposure to a regular (low mark) online quiz instrument has a significant and positive effect on student learning as measured by an end of semester examination”.

Another good resource on the benefits of assessments to check out is the white paper, The Learning Benefits of Questions. In it, Dr. Will Thalheimer of Work-Learning Research reveals research that shows that questions can produce significant learning and performance benefits, potentially improving learning by 150% or more. The white paper is complimentary after registration.

John Kleeman will discuss benefits and good practice in assessments at the 2015 Users Conference in Napa Valley, March 10-13. Register before Dec. 17 and save $200.

Can online quizzes before lectures increase reading by literature students?

John Kleeman HeadshotPosted by John Kleeman

It’s often suggested in higher education circles that computer-assisted assessment is more useful in scientific subjects than in the humanities.

I’d like to share a counter view from some research by Dr Judith Seaboyer at the University of Queensland. She presented a paper at the 2013 International Computer Assisted Assessment conference about how computerized quizzes can help greatly in teaching English Literature.

One challenge of Literature courses is ensuring students read required texts in advance of lectures: Sometimes students struggle to make time for necessary reading, but if they fail to do it, they will likely struggle later on in essays and exams.

Dr. Seaboyer’s solution? Require students to take an online quiz before each lecture. Students must complete the quiz before midnight the night before the first lecture on a text. The quiz, which includes 6 questions chosen at random from a pool of about 15, gives a small amount of course credit. The questions are as Google and eBook search-proof as possible: using different words to those in the text, so they require real reading and understanding.

Here, for example, is a question about Ian McEwan’s Atonement:

Where does Robbie notice a human limb, the memory of which will return to haunt him?

(a) in the fork of a tree

(b) in a Joe Lyons tea house

(c) on the beach at Dunkirk

The right answer is (a), but this would not be easy to identify by searching, as the book mentions a human “leg” not a “limb” and the other answers are plausible. Unless you’ve read the book recently, you will struggle to answer.

Students reported that the online quizzes motivated them to complete assigned reading before the lecture as can be seen in the survey result below:

"The online quiz motivates me to complete assigned reading before the lecture" Mean 4.31 on Strongly Agree to Strongly Disagree Likert Scale

Dr. Seaboyer’s  preliminary research suggests that 83% of first year English students read at least 5 out of 6 books in a course where quizzes were used as against around 45% in a control group.

I’ve seen other examples of quizzes encouraging learners to access learning material that they might otherwise put off until later, and I’d encourage others to consider this approach.

To quote Dr. Seaboyer:

“Computer-assisted assessment can result in more reading and persistent, careful, observant, resilient reading that leads to critical engagement.”

She also believes that this could also apply across a range of other disciplines as well as Literature.

How much do you know about assessment? Quiz 1: Cut Scores

Posted by John Kleeman

Taking quizzes is a fun way to learn. One of the Questionmark blog’s most popular-ever entries was Howard Eisenberg’s Take our Quiz on Writing Good Test Questions, back in 2009.

Readers told us that it was instructive and engaging to take quizzes on using assessments, and we like to listen to you! So here is the first of a new series of quizzes on assessment topics. This week’s quiz is on setting a cut score (pass score).  The questions, written for us by Neil Bachelor of Pure Questions, are about what to do when designing a diagnostic test for safety procedures.

We regard resources like this quiz as a way of contributing to the ongoing process of learning about assessment. In that spirit, please enjoy the quiz below and feel free to comment if you have any suggestions to improve the questions.

Now on to the quiz! Be sure to look for your feedback after you have completed it!

Embedding Questionmark Assessments in Google Wave

Screenshot of an embedded assessment in Google WaveEmbed a Questionmark Perception assessment, survey or quiz inside your Google Wave profile.

  • To see how this would look, see a snapshot of an assessment embedded into Google Wave.
  • Check out this How-to on our developer Web site.
  • Google Wave is an online tool for real-time communication and collaboration.  Embedding an assessment into Google Wave may be useful if you want to ask the members of your Wave to complete a quiz or  simply fill in a survey. The results can then be analyzed and reported on from Perception.

How to Create and Manage Questions in Questionmark Perception

joan-small1Posted by Joan Phaup

Questions are the building blocks for all sorts of assessments, from surveys and quizzes to tests and exams. Whether you create items within Perception or import them from other programs, organizing them by topics helps you manage item banks effectively. Topics can be aligned with learning objectives — a big advantage in identifying knowledge gaps, measuring skills and prescribing learning and development plans.

The following feature tour  shows how to set up a topic structure and then add questions to it. It shows how to edit questions, add graphics and fine-tune the layout before previewing and saving questions. If you are not familiar with Questionmark Perception, this video will give you a basic understanding of how to create questions and put together a well-organized assessment.

You will find other simulations in Questionmark’s Learning Cafe, which includes resources about best practices as well as tutorials about the Questionmark Perception assessment management system.