PISA – the case for ‘So what?’

There’s masses of information about the OECD PISA studies out there. Whatever your opinions on the matter you can find evidence to support them.

My view on PISA has always been “So what?”.

The number of changes between each PISA cycle prevent the data being used in a simple way. There is perhaps some merit in looking at the performance of a single country over time, though even the benefits of that are mitigated by changes that can occur in a country over any given period. Like all tests the studies provide information. However, I have a view that PISA has been over used given the information that it actually conveys about a nations education infrastructure.

There is a question I have always had about PISA that leads to that “So what?” stance. Putting aside for a moment the whole “education for educations sake” argument (which I do subscribe to) there is a reason we spend billions on education. It’s because we assume (and I also subscribe to this) that a better educated population will lead to a more productive workforce. Suggesting such a thing does not make me a fat-cat capitalist enemy of the people. It’s just that I want some productive people around in a few years time to be creating enough wealth to pay my pension. I think it is important that our education system is at least in part aligned to that need.

If PISA is really useful in helping understand how well our education system is performing in this regard we would expect to see some form of correlation between the studies and national economic performance. We might expect to see rising GDP for those countries where their PISA score (or ranking is rising). We might expect to see countries with similar PISA scores experiencing similar economic performance. Lets have a look.

Firstly, a warning. I am looking at one statistic at one point in time, using an incomplete set of data in a way that it wasn’t intended to be used (some might say that PISA was destined to only be used as it was not intended to be used, but that’s another blog). What we have here can only be an indicator of where much, much more work would need to be done.

I have the 2009 PISA Overall scores for 25 European countries. These have been matched up against the July 2013 Youth Unemployment Rates for those countries (which is why the data is partial, I don’t have the unemployment rates for Albania, Azerbaijan, Greece, Iceland, Kyrgyzstan, Liechtenstein, Montenegro, Norway, Serbia, Switzerland and Turkey). I have used the 2009 PISA figures as heuristically you might expect that the academic performance of 15 y/os in 2009 might be an indicator related to current youth unemployment rates.

Here is the scatter graph.

The red dot is the UK. The yellow dot is Germany and the dot right at the top is Spain.

Now, if there was a relationship between the PISA score and Youth Unemployment we would expect a line drawn through the points to slope downwards, as the line of best fit on the graph does indeed do. However, the R-squared value is very low. A figure of 1 would indicate that the two variables are perfectly correlated and a figure of 0 would indicate that they are completely unrelated. 0.028 would indicate that there is not a lot of correlation between the figures.

OK, this is one stat, one time period etc etc. All the things that I would criticise any set of statistics for. What I have written above is insufficient evidence to draw any conclusions about PISA from. I do, however, think it suggests that the linkage between measurable academic output and economic performance is not straightforward. As PISA is used as a reason to make considerable changes to education systems I would expect there to  be a valid (and statistically sound) case put forward as to why, economically, this should be done.

I’ve not seen such a case so far and so will continue to maintain my “So what?” stance.