Are we wasting too much COVID-19 vaccine?

Today comes news that we’ve wasted about 15.1 million doses of COVID-19 doses here in the US. Is that a lot, or a little? How does it compare with history? Is the difference, if any, just random or is it real, i.e., statistically significant?

Scare headlines from the mathematically illiterate

Today I saw a rather nice tweet pointing to a scary-sounding headline about vaccine wastage, and then quite sagely pointing out that this seems to be less than historical vaccination campaigns:

I dunno much about this guy, “b-boy bouiebaisse”. However, I do love bouillabaisse and her American cousin cioppino. So he’s obviously a man of considerable taste!

More importantly, he’s a man of considerable common sense: when somebody gives you a percentage, you ask “Percent of what?”; when somebody gives you a large number, you ask “Compared to what?”

Every. Single. Time.

Ok, let’s dig into the crazy

First up is the source of the scare headline, from NBC News ^[1]: 15 million doses “thrown away”, with the first-world guilt-inducing subtitle “As countries across the world clamor for vaccine doses, U.S. pharmacy chains and state health departments have thrown millions away.”

Sheesh. Where do we even start with how wrong this is?

• The title is totally loaded language: “thrown away” makes it sound like somebody deliberately trashed doses. Gee… maybe some of them expired? Maybe sometimes they couldn’t use the whole vial because there weren’t people waiting?
• The subtitle is a deliberate stilletto flash, slyly implying that the US is wasting valuable resources while the rest of the world starves. While this is arguably a historical pattern, it’s just gratuitous here. Sort of emotional click-bait.
• Then, under the obligatory-but-irrelevant picture, in tiny type: “The number of discarded doses is still a small fraction of the total doses administered in the U.S.” So which is it: are we wasting huge quantities, or being efficient with only a small fraction of waste? You can’t have it both ways, but here it is in consecutive sentences!

Having read only the headline, subtitle, and irrelevant picture caption with exasperation, I did not have much hope for the rest. It sort of dances around, citing all sorts of irrelevancies, without really telling you the 3 things that are actually relevant:

1. What are doses that are “wasted”, i.e., disposed of, as a fraction of the total? Without looking at this, you can’t really judge whether the number is large or small.
2. How does that fraction compare with previous vaccine campaigns? Without looking at this, you can’t tell whether you’re doing any better or worse than could be expected.
3. Is the difference between those 2 fractions statistically significant? Without looking at this, you can’t tell if the difference is real, or arose solely by chance.

Instead, they use loaded language that is less reportorial, and more accusatory:

Pharmacies and state governments in the United States have thrown away at least 15.1 million doses of Covid-19 vaccines since March 1, according to government data obtained by NBC News — a far larger number than previously known and still probably an undercount.

Then a few paragraphs later, they admit they’re undermining their own case, in a fit of intellectual incoherence:

The number of discarded doses is still a small fraction of the total doses administered in the U.S.

Anecdote after anecdote, they tell numerous tedious stories to avoid doing anything useful. Irrelevant monthly breakdowns, appeals to guilt, appeals to emotion in general, accusations of lack of planning… feh.

No, never mind the crazy. Let’s run the numbers!

Ok, enough of that trash. Let’s dig into the only 3 relevant facts in the article. Even though the don’t give us the full picture, we can integrate other sources to do so:

• The US has distributed 438 million doses.
• The US has donated 111.7 million doses to other countries.
• The US has disposed of (not “wasted”) 15.1 million doses for various reasons.

First, is 438 million doses a lot? Not really: with about 330 million residents, we’d need about 660 million doses even if there were no waste at all. So that’s only about 2/3 of the amount we’ll eventually need. If there are boosters needed – which here at Chez Weekend we’re forecasting at above 80% probability – then it’s only a bit over half as much as we’ll need.

Second, consider the 111.7 million doses donated gratis to other countries. It’s not as though we’re hogging all the output, we’re buying some of it to give away. In fact, the US is giving away a lot of the purchased doses, leaving less than what’s needed to fully vaccinate the US, let alone give boosters, which is the opposite of what was implied by the article above:

Third, look at the 15.1 million doses that somehow were not put in arms. Could be freezer failiure, cracked vials, multi-dose vials where you don’t have enough patients to use them up in time, and so on. Nobody’s “wasting” anything. The percent not put in arms is:

So… there was about 3.4% “waste”. That doesn’t sound too bad, right?

Comparison with other vaccination campaigns

How have other vaccination campaigns fared in the past? That’s the right comparison, not an ideal of 0% wasted.

Starting from a popular article ^[2], we dug in a little bit for primary sources. The World Health Organization studied this problem before COVID-19, and published a concept note ^[3] with a bit of amusing mathematics to make probabilistic estimates of the amount of vaccine wastage, due to various causes. While that document is more of a proposal to do better with statistical models, their general estimates are shown in Table 1, reproduced here:

• In mass vaccination campaigns, the range is from 5% – 20%.
• The bigger the vial and the more stringent the cold chain requirements, the more the waste.

The COVID-19 mRNA vaccines have the most onerous cold chain I’ve ever seen, and are in multi-dose vials (as far as I can tell, currently 6/vial for Pfizer and 15/vial for Moderna). That means the appropriate scale of comparison is either row 2 or 3 of the table, i.e., about 10% – 15% wastage during a vaccine campaign.

What we actually observe here is 3.4% wastage of COVID-19 vaccines. In other words, we are doing about 3-5x better than historical campaigns have done, wasting significantly less vaccine! In fact, we’re below the 5% historical waste level even for single-dose vials.

This is the opposite of the conclusion the flashy article above would have invited.

Is that just chance, or is it really better?

Ok, fine. But is that just due to chance, or are we really doing better? This is what statistics is for: given some evidence, what should we believe about it?

It’s the usual proabilistic model. Let $p$ be the probability a given dose gets “wasted” (for whatever reason, legitimate or not). Then:

• For each vial, it’s like a coin flip with a loaded coin that has probability $p$ of coming up heads – a Bernoulli distribution.
• Then count the number $k$ that are wasted out of $N$ total – a binomial distribution.
• If you observe $k$ out of $N$ are wasted, you infer approximately $p = k/N$. However, in a Bayesian framework, if you assume a uniform prior on $p$ (we make no committments of any sort about the potential values of $p$, beyond that it’s in $[0, 1]$), then the posterior distribution is a Beta distribution, because binomial and Beta are conjugate priors.

Now, for our case, $k = 1.51 \times 10^{7}$ and $N = 4.38 \times 10^{8}$. How does the corresponding Beta distribution look, as calculated by a little R script? ^[4]

• As you can see, it is very strongly clustered around the approximate answer above of 3.4%.
• The 95% confidence interval on the probability a vaccine dose is wasted is 3.446% – 3.449%.

So we’re very sure that the probability of wasting a dose is around 3.4475%.

But let’s do the formal hypothesis test anyway: how far out in the tail are the values shown as wastage rates in the past of 5%, 10%, and 15% that we should compare against? Let’s see:

> library("plyr")

> k <- 15100000; N <- 438000000
> ddply(data.frame(WasteRate = c(0.05, 0.10, 0.15)), "WasteRate", function(df) {
    data.frame("p.Test" = pbeta(df[1, "WasteRate"], k + 1, N - k + 1, lower.tail = FALSE))
  })
  WasteRate p.Test
1      0.05      0
2      0.10      0
3      0.15      0

Effectively, here we’re looking at the graph above and asking what’s the probability of getting a waste rate as high as 5% – 15% as seen in the past? In all cases, the probability is basically 0! The uncertainty around the probability of wasting a dose ($p$) is so strongly centered around 3.4475% that there’s no chance in the world that we’re actually as wasting as the past studies, and saw this value by chance.

In other words, the idea that the waste of COVID-19 is less than past studies is wildly statistically significant: the result is real, not by chance; we’ve really done better.

The Weekend Conclusion

Oy, where to start?

The article is pretty trashy: it makes insinuations that doses are being wasted needlessly and flagrantly in the US when the rest of the world is in need. The facts you see above, however, are:

• The US has donated about 25% of its own vaccine supply, and will have to buy more just to get its own population vaccinated, much less give boosters.
• The “waste” rate is 3.4475% (95% confidence limits: 3.446% – 3.449%).
• This compares favorably with the WHO-documented historical waste rates for demanding cold chains and multi-dose vials of 10% – 15%.
• This result is robustly statistically significant: there is not a chance in the universe that we’re really as profligate as historical campaigns, but happened to get 3.4% just by luck. We really are doing better.

And that, boys and girls, is why you never trust the news media when they dance around something to avoid quantitative reasoning. They’re frequently wrong, reflecting the anxieties of the age rather than reality.

Notes & References

1: J Eaton & J Murphy, “15 million Covid vaccine doses thrown away in the U.S. since March, new data shows”, NBC News, 2021-Sep-01. ↩

2: S Schiffling & Liz Breen, “COVID vaccine: some waste is normal – but here’s how it is being kept to a minimum”, The Conversation, 2021-Jan-11. ↩

3: World Health Organization Immunization Programmes Staff (Contact: Souleymane Kone, of WHO Supply Chain Group), “Revising global indicative wastage rates: a WHO initiative for better planning and forecasting of vaccine supply needs”, World Health Organization (Concept Note), 2019-Apr-08. ↩

4: Weekend Editor, R script to plot posterior Beta distribution of probability of a wasted dose, Some Weekend Reading blog, 2021-Sep-02. ↩