Saving time and money with diagnostic assessments

Joan Phaup HeadshotPosted by Joan Phaup

It’s always a pleasure to talk with customers about the ways in which they use assessment to meet their business needs. My recent conversation with Dr. Steve Torkel, Director of Evaluation and Assessment at PwC, was no exception.

I asked Steve about the case study he and his colleagues, Sean Farrell, and John LoBianco, will present at the Questionmark 2013 Users Conference in Baltimore March 3 – 6. It’s called The Half-Time Strategy: Saving Your Organization Time and Money with Diagnostic Testing – and I wanted to get the story behind that intriguing title.

Could you tell me about your job role at PwC and your involvement with assessments?

Dr. Steve Torkel

Dr. Steve Torkel

I am the U.S. evaluation and assessment leader. I lead a team that is responsible for creating assessments for a variety of programs across the U.S. firm and evaluating the impact of training courses throughout the firm.

Your session will focus on how diagnostic assessments play an important role at PwC. Why is that topic important to you?

Diagnostic assessments help us run training like a business. It helps us put the right people in the right program at the right time, which is very different from saying, “We have these training programs; go to them.”  It’s being respectful of our staff’s valuable time. When I say diagnostic assessments, I mean before someone participates in a training program — a face to face or e-learning program — we give them the opportunity to in essence “test out” of it. If they know the information already, why should they take the time to participate in the program?

Your session title talks about a “half-time strategy.” Can you elaborate on that?

Imagine a football game, just before the end of the first half. The score is 42 nothing. Knowing that the team with no touchdowns is not going to come back from 42 nothing and win the game, wouldn’t you just as soon end the game right then?

We call our approach to diagnostic assessments a half-time strategy because) we split the assessment into two halves. If someone performs poorly on the first half, we end it and don’t give them the opportunity to take the second half. They would be wasting their time and perhaps getting more frustrated. It benefits them to stop: It saves them time and frustration. It also has benefits for the firm: We are looking at the time people spend. If we can drive down the time people send on an initiative and still maintain quality, it’s good for everyone.

What do you expect your listeners to take away from your session?

I expect them to take a way a couple of things. One is to learn how to use this approach to help to run training like a business. Another is for them to realize that they can still deliver a quality assessment if they go through the right procedures. Half the time does not equal half the quality! We are saying that if you go through a very structured procedure, you can deliver a high level of quality and save time for your learners.

We’ve created a spread sheet that helps us estimate various factors, for example how many questions we are going to use on an assessment, how long it will to take people to complete it or how much time it will save. We use this information to determine if a diagnostic approach is going to make sense. If it’s only going to save a little time, it doesn’t make a lot of sense. So we’ll be sharing this interactive way to help people figure out if a diagnostic test makes sense in a particular situation.

What are you looking forward to at the conference?

Networking with other assessment leaders at organizations, to see how they are running their assessments – and to find out if anybody has any other business-oriented approaches to assessments. For me, that’s really the key! I’m looking for how people are using assessments to run their businesses more efficiently.

Register for the conference by January 18 to take advantage of early-bird discounts. Check out the conference agenda.

 

Assessment types and their uses: Formative

Posted by Julie Delazyn

Assessments have many different purposes, and to use them effectively it’s important to understand their context and uses within the learning process.

Last week I wrote about diagnostic assessments, and today I’ll explore formative assessments.

Typical uses:

  • Strengthening memory recall and correcting misconceptions
  • Promoting confidence in one’s knowledge
  • Enhancing learning by directing attention to and creating intrigue about a given subject
  • Measuring learners’ knowledge or skills and telling them how they’re doing
  • Giving learners search and retrieval practice and prescriptive feedback

Types:

  • Quizzes
  • Practice tests and exams
  • Self-assessments

Stakes: low

Example: An instructor gives a quiz to help reassure students that they’re actually learning — or alert them that they are not learning and provide feedback to correct any misconceptions. Students can use this feedback as a study guide to understand where they’re going right and where they’re going wrong. Students also benefit from the search and retrieval practice they’ve had while taking the quiz – which can help them remember the material in the future. Formative assessments give instructors a way to ask students: “Did you get that?” Sometimes, a series of quizzes is used to collect data that contribute to overall grades – but generally, formative assessments serve as check-ups on learners’ understanding and guideposts for further progress.

For a fuller analysis of assessments and their uses check out the white paper, Assessments Through the Learning Process. You can download it free here, after login. Another good source for testing and assessment terms is our glossary.

In the coming weeks I’ll take a look at three remaining assessment types:

  • Needs
  • Reaction
  • Summative

Assessment types and their uses: Diagnostic

Posted by Julie Delazyn

Assessments have many different purposes, and to use them effectively it’s important to understand their context and uses within the learning process. I’ll explore each of these five key assessment types over the next few weeks:

  • Diagnostic
  • Formative
  • Needs
  • Reaction
  • Summative

Let’s start with diagnostic assessments.

Typical uses:

  • Identifying the needs and prior knowledge of participants for the purpose of directing them to the most appropriate learning experience
  • Determining knowledge and identifying skills gaps and needs
  • Placing learners in appropriate courses and tailor instruction to their needs
  • Providing instructors and mentors information on a student’s abilities
  • Giving feedback to participants and providing recommendations for products, services and/or learning activities
  • Setting benchmarks for comparison with post-course tests
  • Analyzing personality traits in order to predict behaviors
  • Creating intrigue about the content of a learning activity, which can in turn actually enhance the learning experience

Types:

  • Pre-tests
  • Placement tests
  • Self-diagnostic tools
  • Personality assessments

Stakes: low/medium

Example: A diagnostic assessment might report that a learner has mastered every competency in using Microsoft Word but can only perform 50 percent of those required to use Excel. The results of the assessment would prescribe a course on Excel. In addition, a diagnostic assessment can help place students within suitable learning experiences by asking questions such as, “Do you prefer instructor-led training or online training?”

For a fuller analysis of assessments and their uses check out the white paper, Assessments Through the Learning Process. You can download it free here, after login. Another good source for testing and assessment terms is our glossary.

Tune in next week for a post on formative assessments.

Diagnostic tests that measure conceptual understanding

Posted by John Kleeman

I’ve just read a thought provoking article on diagnostic tests written by Simon Bates and Ross Galloway from the University of Edinburgh Physics Education Research Group and published by the UK Physical Sciences Centre (see the article at pages 10-20 here).

The authors are particularly concerned with diagnostic tests that measure conceptual understanding and identify mis-conceptions. So rather than testing for facts or knowledge or particular skills, their interest in diagnostic assessments is primarily around whether students understand some key concepts in the Physical Sciences. If students don’t understand them, they as instructors need to correct this in their teaching and feedback.

The article gives examples of use of diagnostic tests and also gives some good and detailed guidance on how to construct them, including which statistics to look at for good results. They recommend (as proposed by other authors in the Physics Education Research literature) a p-value or difficulty index of 0.3 to 0.9, a discrimination index of 0.3 or better or a point biserial correlation of 0.2 or better, and a reliability index of 0.7 or better.

They also explain how to write questions that test why people don’t understand something as well as what they don’t understand. And they give the example below (from the Lawson Classroom Test of Scientific Thinking) as something they have used in their teaching. Here is a what-why question, which asks for a fact and also asks why that fact is the case.

Picture of what-why question

Bates and Galloway report that the first, “what” part of the question is answered just as well by students coming into university as those who have completed their first year at university, but that there is significantly better performance in the “why” part by those who’ve been at university for a year.

Getting to the root of learner misconceptions is a key challenge for all of us in learning and assessment, and I recommend this article as a good read.

Helping test publishers profit from their quizzes and tests

Joan Phaup

Posted by Joan Phaup

Our release today of  “Technologies for Selling Tests” gives professional associations, textbook companies, awarding bodies and other test publishers a way to streamline individual and bulk sales of their online quizzes and tests.

This new web-based service is available to users of our hosted and subscription solutions. If you are a test publisher and want to sell your tests to large institutions or make quizzes, diagnostic assessments and other study aids available on a charge-per-use basis, “Technologies for Selling Your Tests” could be the right solution for you. You can use it for consumer purchases via e-commerce or bulk purchase by large institutions.

This solution offers way to improve the return on a test publisher’s investment in creating  and maintaining valid and reliable assessments. It’s all based on Questionmark Perception’s out-of-the-box functionality, and it’s available for you to try out any time you like.