The Ladder

I have been prompted to write this after reading @tombennett71’s latest blog post which you can find here. To be honest, if you only have time to read one thing this morning, pop off and read that. Comparatively speaking this is just a few paragraphs of self-serving, maudlin drivel.

Many years ago, before FFT became the de facto school standard (or even actually existed), a colleague and I set out to use the shed loads of data that the school had collected over the years. We had already put our Praise and Concern system online, removing 13,000 sheets of quadruplicate paper a year from the system (every member of staff got to take home a now unnecessary filing cabinet) and decided we could do a similar job with grades.

Briefly, here’s what we did. We had for every student who had been through the school in the past 15 years every piece of assessment data that had been collected – NFER CAT score, KS2, KS3, GCSE, Teacher assessments. We had effort grades, behaviour data and, of course, all those Praise and Concern slips. We created a big database which we used to calculate, for every possible CAT score, the probability of achieving each grade in each subject. We did this on a rolling eight years of data, to balance having enough base data for the outputs to be valid(ish) and ensuring that the data being used was current(ish). We had a 12 form intake, so eight years of data was around 2,500 students per year group, so we had a lot of data. We did this for every subject. Clearly we had a lot more data for English, Maths and Science than we did for, say, Music and RE, but we had enough to make it usable for most CAT scores for most subjects.

We were very clear that we did not want to have just one number for a student. We did not want to say to students “this is your grade”. We wanted to show that for every student in every subject, all things are possible. And, in most subjects, in this school, with these teachers, for most CAT scores, all things were. In only a very few cases were there never Grade A’s. So we presented this as a stacked bar chart and described it as a ladder. Each students set of ladders were different and we used them to show students that all outcomes were possible. We were also able to use the effort data we had to show that the better the effort, the better the outcome.

For a while it worked. I can’t say that it produced demonstrable improvements in outcomes, because I don’t have the data to prove that. For me, as a concept it worked. So, what stopped it? Two things. Firstly, we had to produce reports to parents. There was no space on a one page data report to put all this information. So we compromised. You might call this our Mosquito moment. We agreed to aggregate all the data together and produce for every child for every subject the dreaded “Expected Minimum Grade”. One value. A single letter, condensing thousands of data points into one value. And we used it to describe a childs prospects. The ladders were rarely seen or used again afterwards. They were too time consuming to produce and to explain. Everyone loved the single value.

This is the first time I have written his down, and I have tears in my eyes as I type. It was a compromise. No one died (I think), but I know (I think) that if I were asked to do the same thing today, I would say no.

The second thing? Well, that was FFT. It became ubiquitous. I have no beef with FFT data as such. Data is neutral. To paraphrase, Data doesn’t crush dreams, the use of data crushes dreams. Every time a child is told “your grade is…” or “your target is…” a fairy dies. And its no use telling me that is not the way FFT is supposed to be used.  I KNOW THAT!! I have a degree in Statistics. I understand how the data is produced and how it can validly be used. Not everyone does. Certainly it is true that few students will understand the nuance between target and estimate.

This is probably the point at which I should say who I blame for this. Truth is I don’t blame anyone. In the same way I created those ladders with the best interests of our students at heart, so did FFT create their data. So did schools who brought it in for their students, so did teachers who used it and so did parents who found it helpful. So did governments who used it for accountability purposes. There was no evil intent here, no malign design. But it is a house of cards built on quicksand.

Repeat after me. “You can’t describe a child with a single letter”. Louder. “YOU CAN’T DESCRIBE A CHILD WITH A SINGLE LETTER”.

So, if you’ve stayed to the end, thank you for listening. And now you should get off and read Toms blog post.

Advertisements