Correlation, causation, food banks and benefit cuts

"Spurious correlations" is a website created by the management consultant Tyler Vigen which delivers, magnificently, what its name promises. It shows a graphical correlation between the number of Americans who drown in swimming pools and the number of films in a year starring Nicolas Cage. 

The site features evidence that the number of murders by "steam, hot vapours and hot objects" moves in tandem with the age of Miss America. One particularly close relationship is the number of civil engineering doctorates awarded and the US per capita consumption of mozzarella cheese.

But the crowning glory of the website is a chart showing how the use of UK food banks shot up as the British state imposed benefit cuts on the least well off.

Actually, it isn't. There's no such chart adorning Vigen's site. But government ministers seem to feel this ought to be on his list of laughable correlations.

The Trussell Trust has announced that it provided Britons with a record level of emergency food supplies - 1.3 million - in the year to March 2018. That's up from just 3,000 in 2006. The number of supplies started rising rapidly around 2012, when government benefit cuts, enacted by the previous chancellor George Osborne, started to bite.

The charity says that the explosion in demand really is driven by benefit cuts, in particular the punitive new "sanctioning" regime, where benefits can be cut or withheld as a punishment for, say, missing appointments or failing to apply for enough jobs.

But the government denies it. "It's wrong to link a rise [in food bank use] to any one cause," said a spokesperson for the Department for Work and Pensions (DWP) when asked about the latest Trussell figures.

But what other "causes" could there be? The DWP does not elaborate, but a former minister at the department, Lord Freud, hypothesised a few years ago that it was actually a rising number of food banks which was stimulating demand. "Clearly food from a food bank is by definition a free good and there's almost infinite demand," mused the former investment banker.

Jacob Rees-Mogg advanced a related theory on his LBC radio show last year, claiming that food bank usage had shot up because ministers had changed policy and allowed job centres to inform people that the charitable resource existed: "The real reason for the rise in numbers is that people know that they are there and Labour deliberately didn't tell them," he said.

Maybe ministers are thinking of such explanations when they object to charities that suggest a link between their benefit cuts and soaring food bank use. Many will be tempted to take the analysis of charities, which actually distribute food to people and talk to them, more seriously than the airy theorising of various well-heeled Conservative politicians.

But we should respect the underlying serious message of Vigen's website. Correlation really doesn't automatically equate to causation and we should beware of taking charts at face value. Proving causation beyond doubt in the social sciences is nigh on impossible, but, at the very least, an observed association needs supporting evidence before we can talk confidently of one thing causing another.

So what evidence do we have in relation to food bank usage? Two academics from the University of Kent, Owen Davis and Dr Ben Baumberg Geiger, analysed pan-European survey data on food insecurity. They found that insecurity rose across Europe in the wake of the global financial crisis and associated recessions.

And the rise was especially stark in the UK.

This in itself should raise warning signs over the Freud/Rees-Mogg argument that people are using food banks not because something has got worse in their circumstances, but simply because they're there.

But what about the benefits link? Other researchers - Rachel Loopstra, Aaron Reeves, David Taylor-Robinson, Ben Barr, Martin KcKee and David Stuckler - have found that food banks are more likely to open in areas with higher unemployment rates and where local authorities have had greater aggregate cuts in welfare spending. There is also an association between higher local benefit sanctioning rates and the amount of local food parcels distributed. Analysis of the types of people who attend food banks shows they are much more likely to be on benefits or with low and precarious incomes.

The evidence is not conclusive, but it is highly suggestive. When government ministers and Conservative Party pontificators can produce a similar weight of evidence to support their own pet theories they will merit a hearing. Until then, we can file their views in a cabinet labelled "politically-motivated reasoning".

Now perhaps that would make for a good website…

The grammar school fallacy

No fallacy dominates popular attitudes to education like post hoc ergo propter hoc ("after this, therefore because of this"). Y follows X, so X must have caused Y.

Consider three common lines of reasoning. Much of the British elite went to Oxford and Cambridge, therefore Oxford and Cambridge must be the best universities and ambitious youngsters should therefore strive to get into those institutions. University leavers tend to get higher paying jobs, therefore a university education must give young people the skills necessary to secure such employment, and therefore almost all young people should go to university.

And because grammar schools produce children with excellent exam results, that must be because grammar schools are excellent schools, and we should wish to see more of them. Post hoc ergo propter hoc.

But it ain't necessarily so. What if the 18-year-olds who go to Oxford and Cambridge and other Russell Group universities are already bright? Perhaps those particular institutions aren't really adding very much in academic terms at all.

What if university leavers already had many of the skills required to thrive in the modern workplace before they arrived? Perhaps a university education doesn't actually confer those abilities after all, and that from an economic perspective, it's a waste of resources to cram ever more young people through this particular form of higher education.

To identify whether a particular factor causes an outcome, we need to consider the counterfactual. In other words, to ask what would have happened in the absence of that factor? Establishing counterfactuals is tough but, to be credible, an analysis has to at least attempt it.

Now let's consider the case of the 164 grammar schools in England, which select girls and boys at age 11 on the basis of their academic ability. There's no question that the 167,000 students who attend them get better GCSE and A level results on average, relative to the roughly three million pupils who attend nonselective state schools. But is that because of the quality of the teaching at the schools themselves and their selective mechanisms? Or is it because of the pre-existing abilities of the children who attend them, perhaps because they come disproportionately from well-off families? What's the counterfactual? Might those children perhaps have gone on to do well anyway in another school environment, such as a comprehensive? Research from Durham University suggests a definitive answer to that.

When the academics make statistical adjustments for the prior abilities of students and the wealth of their family background (grammar schools do tend to be dominated by the middle classes), they find little evidence that these schools add much.

However, there is some pretty compelling statistical evidence that poorer children in regions such as Kent, where grammar schools are still relatively common, have worse academic results than one would otherwise expect. This suggests brighter children with wealthier parents tend to drive up overall standards in comprehensive schools. But the corollary of this is that the presence of grammar schools is indirectly harmful to the wider local schools system. "Grammar schools in England endanger social cohesion for no clear improvement in overall results," concludes Durham's Professor Stephen Gorard.

When she became Prime Minister in 2016, Theresa May promised she would allow the establishment of new grammar schools. For the first time since the mass comprehensive conversions of the 1960s, the government planned to encourage them. The evaporation of her authority after last year's election fiasco has forced May to retreat from that. Yet it remains Government policy to allow existing grammar schools to expand, in itself a significant departure from the cross-party policy consensus of the past 40 years.

Many of those who want to expand academic selection at age 11 (though not all) are inspired by decent motives. They not only look upon the good academic results produced by grammar schools and wish to proliferate that success, but also observe that the post-war era of expanding grammar schools was also an era of rising social mobility, when it felt like talented young people from modest backgrounds could "get on".

Yet they may have been tripped up by another statistical fallacy: confusing correlation with causation. Was the acceleration of social mobility of this era due to the introduction of grammar schools in the 1944 Education Act? Or more "room at the top" in British economic life, as white-collar work and the professions expanded rapidly thanks to rapid technological change? The evidence leans to the latter.

The question of how to resurrect the social mobility of the post-war era is a complex one. But when it comes to education, the most statistically rigorous research we have (drawing on the research of economists, psychologists, neuroscientists and statisticians) strongly suggests the key to boosting lifetime attainment lies in improving (very) early years care and in targeting resources on the least advantaged.  If ministers really care about the life chances of the poorest, they should invest heavily here, reverse the post-2010 cuts to Sure Start funding, and, finally, shed those beguiling grammar school fallacies.