Criterion-referenced or standards-based exams are a positive factor in school performance

Posted by John Kleeman

One of the world’s most impressive testing efforts has just published its latest results.

Every 3 years, the OECD organizes an international testing effort, called PISA (Programme for International Student Assessment) that measures key competencies of 15-year olds in reading, mathematics and science.  In 2009, they assessed 470,000 students in 65 countries and used psychometric techniques to ensure that the test results were comparable internationally and statistically valid. By measuring competencies towards the end of  school life worldwide, and also correlating the answers with demographics, they provide a lot of evidence for what works in education and what doesn’t.

The results for the 2009 tests were published on Tuesday and are available here on the OECD website.  There are more than a thousand pages of analysis and the raw data is also available to download.  As well as being interesting in their own right, the PISA results also show an example of how to analyze results from assessments, particularly the way that they correlate results with demographic data to provide meaning.

The headline news is that the best performance in the world was from the Shanghai region of China, with the strongest countries as a whole being  South Korea and Finland. You may have seen some headlines in your local news. For example, the New York Times reported US Secretary of Education Arne Duncan as saying: “We have to see this as a wake-up call … The United States came in 23rd or 24th in most subjects. We can quibble, or we can face the brutal truth that we’re being out-educated.”  The BBC reports that the United Kingdom has fallen in the years 2000 and 2009 in maths from 8th to 28th and in science from 4th to 16th, with the OECD saying that the United Kingdom school performance is about average in the world and asking “whether the UK thinks that ‘average’ is good enough?” Some other countries are proud: the Edmonton Journal reports that students in Alberta, Canada, performed amongst the best in the world.

I’ve been trying to read between the lines and two things have struck me.

One is that OECD has found there is a substantial correlation between teacher salaries and educational results. Countries that pay their teachers a higher amount (relative to other workers) get better educational results.  As can be seen in the graph below, this has much more impact than total expenditure or class size.

Graph showing correlation of various factors to reading performance (from OECD PISA 2009 results). Copyright OECD

Although OECD is careful to state that they can measure correlations, not cause and effect, it would seem that good, well-rewarded teachers are critical for good learning. I’m reminded of the famous writer Arthur C. Clarke who said, “Any teacher who can be replaced by a machine should be!” Effective teaching gets results.

Another conclusion from the OECD is that in countries that use standards-based external examinations, students tend to do better overall.  The OECD defines standards-based external examinations as being examinations that focus on a specific school subject and assess a major portion of what students studying this subject are expected to know or be able to do and that define performance relative to an external standard, not relative to other students in the classroom or school. Such examinations also have real consequences for the students’ progression or certification in the education systems.

As you can see in the diagram below, these standards-based (or criterion referenced) assessments significantly correlate with better reading performance internationally, though more general use of other standardized assessments has less impact.

Graph showing correlation of various factors to reading performance (from OECD PISA 2009 results). Copyright OECD

It’s great to see testing technology used to inform decision makers in the way that PISA does.

Questionmark’s white paper, Assessments through the Learning Process, includes information about criterion-referenced tests and how to use them.

Leave a Reply