Moderna Covid Vaccine Maybe Not 94.4% Effective; Epidemiologists Can’t Infer Precise Results From Tiny Sample Size
Moderna's Claims For Covid Vaccine Effectiveness Might Be Misleading
WASHINGTON, D.C. (November 16, 2020) - "Moderna's COVID-19 Vaccine Is 94.5% effective" say many headlines, but claiming the precise effectiveness, supposedly accurate to one-tenth of a percentage point, is misleading at best, like claiming that a political poll shows candidate X with 54.3% to 45.7% lead when the poll has a margin of error of +/- 4.0 percentage points, says Professor John Banzhaf.
Michael Mauboussin’s 10 Attributes of Great Investors [Pt.1]
In 2016, Michael J. Mauboussin completed his 30th year on Wall Street. The analyst, who was working at Credit Suisse at the time, decided to celebrate by reflecting on the ten attributes of great investors he had observed over the previous three decades. He published his ideas in a report in August 2016. I've summarised Read More
Moderna's statement, making a claim of effectiveness down to one-tenth of a percentage point, is based upon the report that, among a test population of some 15,000 who received the COVID-19 vaccine, only 5 became infected, whereas of the 90 volunteers in the control group who received a placebo, and presumably were subject to the same exposure risk, 85 became infected.
While it's mathematically true that 85 divided by 90 does equal 94.44%, the number in the vaccinated group who became infected could as easily have been 2 or 3 higher or lower, simply because of random statistical fluctuation plus inherent inaccuracies of tests for COVID-19.
Thus, for example, if the number in the vaccinated group who became infected was 7 instead of only 5, the effectiveness would drop to 92.22%.
Same Claims From Pfizer
And, if the number who actually became infected was 9 instead of 5 - a number which is quite possible given the random uncertainties of medical tests, plus the false negatives of some COVID tests - the effectiveness would drop to 90%, which is the same effectiveness claimed by Pfizer for its new COVID vaccine.
This, of course, doesn't mean that the two vaccines are likely to be equally effective, but rather that a number like 5, out of a total of some 15,000, hardly seems sufficient to draw a reliable statistical conclusion that the vaccine was 94.4% (and not 90% or 98%) effective, argues Banzhaf, a mathematician and creator of the Banzhaf Index.
That's because a change of only a few detected infections in the test group - something which could easily be caused by a random statistical fluctuation and/or by inherent inaccuracies of the test for COVID - could cause a significant change in the number of volunteers who apparently became infected, and therefore in the claimed effectiveness figure itself.
That's why most epidemiologists and statisticians would probably not draw any firm conclusions from such a small result, especially after factoring in the limited reliability of some of the tests.
Another problem is that the infection rate in the placebo group is less than 0.6% [85/15000 = 0.57%], compared to double digit infection rates now being reported in many states.
Even New York City's infection rate, which is much better than most other jurisdictions, is hovering around 3.0%, which is about 5 times that of the Moderna test group.
So, if these reported figures are accurate, it suggests that everyone in both groups had an exposure to the coronavirus which was - for some reason which was either not known, or not revealed to the media - far lower than in virtually any population group in the U.S., and, in many cases, an entire order-of-magnitude lower.
This alone should cast some doubt on the validity and applicability of the results to actual populations in which more realistic exposures levels, as measured by the infection rate, are much higher, and frequently over 5.0%, suggests Banzhaf, who dealt extensively with the epidemiology of tobacco smoke.
In other words, testing a vaccine against disease X, in a population where the infection rate for disease X is an order of magnitude much lower than in many populations, may not provide a realistic or reliable test of its effectiveness in the real world where the infection rate is often many times higher, explains Banzhaf.