What does a 95% effective vaccine mean?

A  Pfizer COVID-19 (C19) vaccine said to be 95% effective in a recently-completed clinical trial is being widely welcomed, particularly by the governments that have already bought it.

However, what does 95% effective mean? Effective against what – C19, the virus that causes C19 (SARS-CoV-2), hospitalisation, severity, death? Further, does 95% effective mean that if 100 people are inoculated, 95 are protected? No, it doesn’t mean that. 

The trial recruited over 44,000 people (mostly white, median age 51, 20%>65) from 6 countries (US, Argentina, Brazil, Germany, South Africa, Turkey), and randomly (and blindly) gave half (22,000) the vaccine, and half a placebo. The study commenced on July 27th, 2020. Anyone (from either group) that became symptomatic for C19 was tested, and if the test was positive, the participant was recorded as a laboratory-confirmed case of C19 (i.e. only clinically symptomatic cases were tested for confirmation, not everyone). At the conclusion of the study, the primary outcome measure was to be a comparison of how many laboratory-confirmed cases of C19 there were in the vaccine vs. placebo groups.

The protocol determined, by statistical models, that once 170 people (out of the 44,000) were laboratory-confirmed cases, the data could be unblinded and the comparison made. This goal was reached on November 14th. There were 162 cases of C19 in the placebo group, and 8 cases in the vaccine group. The interpretation was that the vaccine stopped 162-8=154 cases. The ratio 154/162 is 95%, and there you have it – the 95% effective figure announced by Pfizer.  

While the 162 vs. 8 result may seem powerful, expressing this as 95% effective is not the only way the data could be presented. Remember that the results came from a massive 44,000 people recruited into the study, and therefore the absolute percentages of cases for both groups will be small. For the placebo group, the number of C19 cases was 162/22,000, which is 0.73%. For the vaccine group, it was 8/22,000, or 0.03%. The absolute difference between placebo and vaccine is therefore 0.7%.

Another interpretation: Of the 22,000 people who were not vaccinated (placebo group), 21,838 did not go on to develop C19 (i.e. 99.26%). Of the 22,000 people who were vaccinated, 21,992 did not go on to develop C19 (i.e 99.96%). Thus, over 99% of participants did not develop C19 regardless of whether they were vaccinated or not.

As an aside, 99% may seem a surprisingly high figure for a study based on clinical cases, given that the public message is that we are in the midst of a pandemic. This was over the almost 4 months of the study, and included countries (such as the US and Brazil) that didn’t take significant lockdown measures. Something to ponder. Perhaps the participants who volunteered for the study were not representative of the general population. In which case, the results of the study may not be representative of the general population either.

Finally, we could ask how many people would need to get inoculated for one person to benefit. This is the ‘number needed to treat (NNT)’, and it is 22,000/154 = 143. We have no information on the number needed to harm (NNH), or whether it is either greater than or less than the NNT. If the NNH is less than NNT, then there is net harm. The need to treat 143 people for 1 to benefit, allowing for possible harms in the other 142 people, is another matter.

In any case, all of these ways of expressing the results, some of which may seem wildly contradictory, are mathematically correct. For example, it could either be said that the vaccine had a relative effectiveness of 95%, or that it had an absolute effectiveness of 0.7%. Both are correct, but only one of those numbers has made it to marketing.

What this study doesn’t tell us

The outcome measure was the number of clinical and laboratory-confirmed cases of C19 in each group. One clinical sign was all that was needed for a laboratory test to be undertaken. The clinical signs were any of: fever; cough; shortness of breath; chills; muscle pain; loss of taste or smell; sore throat, diarrhoea or; vomiting. As long as the case was laboratory-confirmed, it could be mild and limited to one of these symptoms.

So, the study does not tell us if the vaccine prevents, or even reduces, deaths. In other words, we cannot say the vaccine is 95% effective against death from C19. The total number of deaths during the study was 6, or 0.01% of the 44,000 people enrolled, which is too small to draw a conclusion (and remarkably small in its own right –  something else to ponder).

As well as deaths, we cannot say the vaccine was 95% effective in reducing:

  • Hospitalisations
  • The severity of C19 symptoms
  • Asymptomatic infection, or 
  • Viral transmission 

Even with further analysis, the study will not be able to answer any of these questions, because it was either not statistically powered to do so, or the data were not collected.

Some inferences might be drawn, and it is likely that Pfizer and the governments that have already paid for their vaccine will lean on that. For example, if the vaccine reduces C19 numbers, then perhaps it will reduce hospitalisation or deaths, but that is an inference and not based on data (it may reduce mild-to-moderate cases, but be less effective in severe cases). Alternatively, reducing C19 severity may lead to more asymptomatic cases that could increase community transmission with the relaxation of public health directives.

It will be a challenge to determine an appropriate public health response as vaccine uptake increases, given the importance of these fundamental and unanswered questions. Questions that are also set to be unasked and unanswered by other drug candidate trials.


Links

Pfizer protocol

Pfizer published results

FDA analysis