Has Vaccine Effectiveness Against Death Been Overestimated?

The ‘stylized facts’ concerning vaccine effectiveness against Covid are as follows. First, the vaccines confer strong protection against infection, which peaks about one month after the second dose. However, this protection then wanes over the following five or six months to as low as 20% (or even lower).

Second, the vaccines also confer strong protection against serious illness and death, and this protection wanes much more slowly. Hence, six months after vaccination, the vaccinated still have substantial protection against death.

While these stylized facts are approximately right, there’s reason to believe that vaccine effectiveness against death has been overestimated. Note: by ‘overestimated’ I simply mean that effectiveness may be somewhat lower than is claimed, not that effectiveness is zero (or even close to zero).

The reason effectiveness against death may have been overestimated was highlighted in a recent study by the U.S. CDC. Using data on a large cohort of Americans, Stanley Xu and colleagues calculated non-Covid death rates (adjusted for age and sex) among vaccinated and unvaccinated persons.

Their findings are shown in the figure below (taken from the Economist). Notice that, on both charts, the light blue bar is much higher than the other two bars, indicating that non-Covid death rates were substantially higher among the unvaccinated.

The researchers interpreted their results as evidence of vaccine safety. (If the vaccines were very unsafe, you’d expect a lot more non-Covid deaths in the vaccine groups.) However, there’s another implication, which the researchers also acknowledge: people who get vaccinated tend to be healthier and/or more risk-averse than those who don’t.

How do we explain this ‘healthy vaccinee’ effect? There are at least two possibilities. First, some people may be too frail to get vaccinated, due to old age or underlying health conditions. Second, some people may just be inherently healthier/more risk-averse, and as a result may be more likely to get vaccinated and less likely to die of other causes.

The CDC researchers actually attempted to control for the healthy vaccinee effect by selecting unvaccinated persons from among those who’d had at least one flu shot in the last two years. The fact that they still observed a difference in death rates suggests the true effect may be even larger.

While the CDC’s results might be good news from the perspective of vaccine safety, they’re bad news from the perspective of vaccine effectiveness.

Suppose you do a study, and find that the Covid death rate is ten times higher among unvaccinated people than among vaccinated people. That difference could be due to the vaccines. But it could also be due to the fact that vaccinated people are less likely to die for any reason.   

Observational studies of vaccine effectiveness do attempt to control for the healthy vaccinee effect, for example by including some measure of health/frailty as another variable in the analysis. However, these kinds of measures may not capture all the relevant differences between vaccinated and unvaccinated people.

Consider Israel. By the start of June, the country had fully vaccinated 55% of its population (and an even higher percentage of adults). Despite this, and despite having gone through two previous waves of Covid, the country saw a major third wave associated with the Delta variant. Here’s the chart of daily Covid deaths:

One might have assumed that natural immunity from the two previous waves, combined with a 55% vaccination rate, would have kept deaths to a minimum. Yet the number of deaths was still substantial, at least relative to the previous waves. This is particularly evident if we look at excess mortality:

Total excess mortality in Israel’s third was about the same as in its first wave, and was actually higher than in its second wave. Now, it’s true that a disproportionate share of Covid deaths were among the unvaccinated. But this is consistent with the healthy vaccinee effect.

Of course, I’m not claiming that a plot of excess mortality in one country constitutes a serious analysis. Mortality would presumably have been higher absent the vaccination campaign. However, it’s hard to reconcile the chart above with claims of, say, 90% effectiveness against death.

To repeat: I’m not claiming the vaccines aren’t effective against serious illness and death; only that their effectiveness (after two doses) may have been overstated. Offering the vaccine to over 50s still makes sense as a way to achieve focused protection.