Negative Vaccine Effectiveness: The ‘Fact Checks’ Were Wrong Last Autumn and They’re Even More Wrong Now

The truth is I’m a bit racked off with both Professor Sir David Spiegelhalter and Tim Harford. I used to go out of my way to hear them on radio; they’re both gifted communicators. Indeed, anyone who can make a documentary about barbed wire a riveting listen – and Tim Harford does just that in his series “50 things that made the modern economy” – deserves all the accolades he gets. However, I wish they’d tidy up a bit behind themselves. I refer, of course, to the furore generated last September by Robert Peston’s tweet and President Bolsanaro’s subsequent use of the information to query whether the truth was that the unvaccinated were less likely to test positive for Covid than the vaccinated.

Why bring it up again now, eight months later? There are several reasons.

Firstly, though in general I’ve stopped listening to or watching any BBC output, I did hear a couple of minutes of the latest BBC Radio 4 More or Less episode where Tim Harford was dragging some unfortunate WHO statistician over the coals for having the temerity to suggest that Germany had done less well than the U.K. in the Covid league of death. It’s not as if he hasn’t got a few things wrong over the past few months himself. People in glass houses, and all that.

Secondly, in a good natured discussion with a doctor friend of mine about the efficacy of vaccines (yes, people can disagree without irrevocably falling out) I mentioned that the last data provided by the UKHSA showing the prevalence of Covid infections in the unvaccinated in comparison to the vaccinated showed that the vaccinated in the 40-49 year-old cohort were more than four times more likely to test positive for Covid than the unvaccinated. She laughed and said she knew all about this. She told me that Professor Sir David Spiegelhalter, no less, Chair of the Winton Centre for Risk and Evidence Communication in the Centre for Mathematical Sciences at Cambridge, had done a thorough demolition of this canard last year. Certainly, to her my argument had been shot down in flames. That annoyed me because we now know Spiegelhalter’s confidence in the efficacy of vaccines against transmission was wrong, yet he’s done nothing to clarify this – a bit remiss for someone whose very job description includes the words ‘risk’ and ‘communication’.

Thirdly, I think the fact checkers need fact checking. Full Fact had done a piece on this which with hindsight we can see was hopelessly premature and yet they too have failed to correct their now apparent error.

Fourthly, I contacted both Professor Sir David Spiegelhalter and the More or Less programme asking whether they would revisit this issue, and while I had a very polite response from the Professor declining the invitation I’ve still heard nothing from More or Less so I thought I’d better set the record straight for them.

Finally, and most importantly, this issue highlights all too well the disservice that the UKHSA is doing us all by stopping the publication of real world data and only publishing weighted data in its (no longer weekly) vaccine surveillance reports. This episode illustrates the deficiencies of weighting and shows how data can be manipulated when they’re obfuscated. Without the raw data, Spiegelhalter and Harford couldn’t have mounted their critiques and I couldn’t have mounted this riposte. Raw data serve all sides.

Let’s go back to the beginning and the furore caused by Robert Peston’s initial tweet that set this particular hare running. This was followed by an attempted takedown by Leo Benedictus of Full Fact after President Bolsanaro of Brazil quoted the same numbers and an article in the Guardian by Professor Sir David Spiegelhalter and Anthony Masters published on September 19th 2021.  In addition, the BBC Radio 4 programme on More or Less covered the story on Sept 15th 2021, featuring a discussion between presenter Tim Harford and mathematician James Ward which concluded that the vaccines were very effective and that there was nothing to see here. The show also informed us that ITV had deleted the Peston tweet.

In their Guardian piece, Spiegelhalter and Masters wrote:

The surprise came from the reported case rate per 100,000 population being higher for fully vaccinated people (1,116) aged 40 to 49 than for unvaccinated (880), which appears to question the effectiveness of the vaccines. But the problem is that we don’t know how many people have not been vaccinated, because we don’t know how many people live in England.

I’ve reproduced below the table from the UKHSA week 36 vaccine surveillance report, which was also shown in Peston’s original tweet. You can see the numbers quoted by Spiegelhalter for the infection rate by vaccine status per 100,000 in the last two columns circled in orange and blue and the raw number of infections by vaccine status in columns two and four. Of course, the number of infections in the vaccinated 40-49 year-olds is the total less the infections in the unvaccinated, 97,881 – 15,106 = 82,775.

The good Professor went on to explain:

PHE uses National Immunisation Management System (NIMS) estimates of 8.1 million adults aged 40 to 49 in England. These numbers depend on GP records and tend to overcount. The Office for National Statistics (ONS) also produces population estimates, based on the 2011 census, migration and death registrations and estimates 7.1 million adults in that age group, a million fewer than NIMS.

Up to September 5th, NHS England reported 6.4 million people in that age group had received a COVID-19 vaccine dose. So the ONS population estimate leaves about 700,000 unvaccinated people aged 40 to 49, while using NIMS means 1.7 million. This is a massive difference.

Let’s see what that massive difference does to the infection rate per 100,000 using the two different population estimates:

The infection rate in the vaccinated doesn’t change whether the NIMS or ONS data are used; it stays at 1,116 per 100,000. All the error is concentrated in the unvaccinated. From the figures above we can see that using NIMS data the infection rate goes from the reported 880 per 100,000 to 2,137 per 100,000, almost 2.5 times what was reported and almost two times the rate of the vaccinated. Simply put, the same number of infections are spread among a smaller group of people if we use the ONS population estimate.

Well, let’s now roll forward 29 weeks to the UKHSA vaccine surveillance report of week 13 2022 to see how these criticisms have played out over time. This was a special week because it was the last time that the UKHSA reported raw data.

Let’s just remind ourselves that this table shows infections by vaccine status for the prior four weeks. You can see circled in blue that 15,437 of the infections in the 40-49 age cohort were amongst the unvaccinated, up 331 (2%) from the 15,106 amongst the unvaccinated back in September 2021. Whereas amongst the vaccinated there had been 238,155 infections (253,592 – 15,437).  That’s 155,380 (187%) higher than in September 2021 (238,155 – 82,775) and, let’s not forget, in the meantime many of this age group had had a third vaccine dose. Isn’t that amazing, the rate of increase in infections was about 90 times greater in the vaccinated than the unvaccinated.

Just to nail the point, let’s also look at the change in infection rates per 100,000. The table below shows the infection rate in the triple vaccinated compared to the not-vaccinated from the week 13 2022 UKHSA vaccine surveillance report.

The triple vaccinated are apparently 4.1 times more likely to test positive than the unvaccinated among the 40-49 year-old cohort (3,957 ÷ 955 = 4.1). Aha! Spiegelhalter, Harford and Full Fact might cry, but again you haven’t corrected for the NIMS/ONS issue. Okay, let’s do that. As you can see below, even if we use the ONS population data, the triple vaccinated were 1.7 times (3,958 ÷ 2,320) more likely to test positive for Covid than the unvaccinated.

In the 29 weeks since the week 36 report, the vaccination rate amongst the 40-49 year-olds is much the same yet the incidence of infection in the triple-vaccinated has increased markedly.

As I mentioned, the UKHSA no longer publishes any raw data so it’s increasingly hard to verify claims, but perhaps it is just worth testing the view expressed by the More or Less team in the September 15th episode, that despite the vaccines not providing complete protection against infection they do still prove very efficacious against death. Well let’s have a look.

Table 7 is reproduced from the week 13 2022 UKHSA vaccine surveillance report. It shows the number of deaths by vaccine status by age cohort. These are raw figures, not deaths per 100,000. As you can see, among those aged over 40, 300 of the 4,005 deaths were of unvaccinated people, that’s 7.5%. Surely, if the vaccines were efficacious you’d expect the unvaccinated death rate to be higher than that of the vaccinated.

Below, I’ve shown the percentage of deaths against the percentage of those unvaccinated using both NIMS and ONS population data.

You can look at the individual age cohorts if you care to but I’ll just comment on the total. At best, using Spiegelhalter and More or Less’s preferred ONS population (which is actually well-known for under-counting the population, even, as the UKHSA itself has pointed out, resulting in some age cohorts having greater than 100% vaccination coverage), 6% of the deaths in the over-40s cohort are not vaccinated yet they only account for 7% of the deaths, hardly a ringing endorsement for vaccination.

If we use the NIMS population data (which the UKHSA calls the “gold standard” for this purpose), 12% of them are unvaccinated, yet only 7% of deaths were of the unvaccinated. Is this why UKHSA stopped reporting the raw data, because they showed that, at best the vaccines appeared to make no difference in the proportion of deaths and at worst appear to show that the unvaccinated are far less likely to die?

Can we expect More or Less to have another look at the data or Professor Spiegelhalter to write another article for the Guardian? No, I don’t think so either. But even they would have to admit that it’s all a bit rum. Still, perhaps Robert Peston or President Bolsanaro will retweet this article.

This article has been updated to correct two calculation errors. The overall argument is unaffected.