The Contradiction in the Covid Inquiry’s Use of Modelling

The Covid Inquiry’s headline figure that an earlier lockdown (March 16th 2020) “would have” resulted in approximately 23,000 fewer deaths in England derives from a counterfactual simulation performed by Professor Ferguson’s Imperial College modelling group, who reported a 48% reduction in first-wave mortality when comparing a March 16th intervention to the actual March 23rd lockdown.

But a close reading of the inquiry’s own evidence shows that this modelling claim has the same structural vulnerabilities that produced some of the worst modelling failures in recent medical history.

However, before we get into the nitty-gritty of these structural problems, it’s worth asking exactly what went wrong with the inquiry’s approach.

The inquiry used Ferguson’s model to create a lockdown narrative: decision-makers acted late; an earlier lockdown would have saved tens of thousands of lives. Therefore, blame is assignable and quantifiable, as the model demonstrates.

Despite the modelling being based on flawed assumptions, not peer-reviewed, unreplicated and mis-specified, the inquiry can assert, as the government did during the pandemic, ‘We followed the science.’

Modellers rely on equations, graphs and computer code to create the appearance of scientific legitimacy. Hallett and her inquiry team placed their trust in this pseudoscientific visual tool, despite its inherent uncertainties. And because they followed ‘The Science’, criticisms have focused mostly on the modelling rather than on the inquiry’s approach.

“23,000 fewer deaths” is powerful because it looks precise, evokes emotion and shapes public memory: people died.

Yet, in Volume II, the inquiry discusses how ministers misunderstood modelling outputs: “Ministers failed to grasp the distinction between model-based forecasts and scenario modelling.”

Scenarios were often wrongly treated as forecasts. Professor Whitty recalled that they “were not meant to be predictions, they were not presented as predictions, but they were often interpreted as predictions” (page 38, volume II).

But now we are supposed to believe model outputs are factual predictions (‘X treatment would have prevented Y deaths’).

How are we supposed to unravel this contradiction?

Scenario models are ‘what ifs’. They explore the consequences of hypothetical actions; they are not predictions of what will happen.

Vallance told the inquiry that modelling can project possible futures, but cannot predict the future, and that ministers repeatedly failed to grasp this distinction.

Vallance and Whitty were explicit: models were not predictions.

Yet the inquiry used Ferguson’s scenario as if it were a factual counterfactual, stating: “A lockdown on March 16th would have resulted in approximately 23,000 fewer deaths.”

The inquiry ignored warnings about data problems: “Modellers had great difficulty in getting clarity on the NHS numbers”. Yet the Inquiry used modelled data as if it were precise estimates — precisely the misinterpretation it was warned about.

Vallance stressed models show possibilities, not certainties. The inquiry, however, chose to use the model as the basis for findings of failure, inevitability and moral condemnation.

The inquiry was also warned that modelling inputs are easily misconstrued; ministers’ confusion was documented and misinterpretation was common. Yet it chose to interpret the most fragile of models as if it were a precise epidemiological and historical truth.

In using models to make strong causal claims, the inquiry ignored uncertainty and alternative evidence about whether lockdowns saved lives. In doing so, it reproduced the exact misinterpretation Vallance and Whitty warned against.

The use of lockdown modelling is wrong because it tries to answer a complicated causal question using tools that are vulnerable to bias, mis-specification and hidden assumptions.

The inquiry did not merely use modelling to justify its claims; it chose to embody the psychological, political and technocratic reasons why policymakers find models so irresistible: the allure of certainty.

Presenting a hypothetical scenario as if it were a verified historical fact is a very risky strategy. Particularly as the claims are scientifically unreliable. Methodologically, the interpretation mirrors previous modelling disasters, demonstrating fundamental misunderstandings of the use of models and their results, which the inquiry vehemently criticises ministers for.

Linking causal statements about the certainties of 23,000 deaths through the use of models is epistemologically unsound. You cannot establish a causal chain of events if you don’t analyse what caused those events. Modelling can’t overcome this shortcoming. The inquiry should know this – the problem is, it doesn’t.

This post was written by two old geezers who don’t like contradictions.

Dr Carl Heneghan is the Oxford Professor of Evidence Based Medicine and Dr Tom Jefferson is an epidemiologist based in Rome who works with Professor Heneghan on the Cochrane Collaboration. This article was first published on their Substack, Trust the Evidence, which you can subscribe to here.