Diagnostic tests that measure conceptual understanding

Posted by John Kleeman

I’ve just read a thought provoking article on diagnostic tests written by Simon Bates and Ross Galloway from the University of Edinburgh Physics Education Research Group and published by the UK Physical Sciences Centre (see the article at pages 10-20 here).

The authors are particularly concerned with diagnostic tests that measure conceptual understanding and identify mis-conceptions. So rather than testing for facts or knowledge or particular skills, their interest in diagnostic assessments is primarily around whether students understand some key concepts in the Physical Sciences. If students don’t understand them, they as instructors need to correct this in their teaching and feedback.

The article gives examples of use of diagnostic tests and also gives some good and detailed guidance on how to construct them, including which statistics to look at for good results. They recommend (as proposed by other authors in the Physics Education Research literature) a p-value or difficulty index of 0.3 to 0.9, a discrimination index of 0.3 or better or a point biserial correlation of 0.2 or better, and a reliability index of 0.7 or better.

They also explain how to write questions that test why people don’t understand something as well as what they don’t understand. And they give the example below (from the Lawson Classroom Test of Scientific Thinking) as something they have used in their teaching. Here is a what-why question, which asks for a fact and also asks why that fact is the case.

Picture of what-why question

Bates and Galloway report that the first, “what” part of the question is answered just as well by students coming into university as those who have completed their first year at university, but that there is significantly better performance in the “why” part by those who’ve been at university for a year.

Getting to the root of learner misconceptions is a key challenge for all of us in learning and assessment, and I recommend this article as a good read.

3 Responses to “Diagnostic tests that measure conceptual understanding”

  1. […] retrieval and perhaps also that diagnoses conceptual misunderstandings if you get it wrong. (See http://blog.questionmark.com/diagnostic-tests-that-measure-conceptual-understanding for an example of this latter type of […]

  2. Suzanne says:

    I need to gather similar diagnostic results, to measure the effectiveness of a new employee on-boarding program. I am looking for some guidance on legitimate sampling group size, and which groups to compare at what points.

    So far I have sent a pre-survey that covers the same content as the final certifcation to employees who have been with the company 2-4 weeks before participating in the new program.
    I am thinking to compare this to the following pre-audiences:
    b)those who start with the New Employee On-boarding Program
    c)sample group that have not done the new program.

    and then do a follow-up to all groups after 90 days.

    Do you think this approach will provide any meaningful daya? Any guidance provided is greatly appreciated, thanks!

  3. John Kleeman says:

    It’s obviously hard to comment without knowing all the facts. But if you have two very similar groups, one of whom does your On-boarding programme and one who doesn’t, then measuring their knowledge and comparing them is likely to be very insightful. Ideally you should select who does the on-boarding and who doesn’t randomly to avoid any possible bias between the groups. For instance if everyone in one city does the programme and everyone in another city doesn’t, they might have different results due to informal learning factors or being different kinds of recruits. If this is not possible, then you should look carefully at other factors that might influence the groups. You could also look at the Success Factor approach (see http://blog.questionmark.com/using-the-success-case-method-with-questionmark-perception), which is a more qualitative way of measuring results that is easy to do and useful. I hope this helps.

Leave a Reply