On the Moderna Monkey Trial of Omicron-Specific Boosters

Somebody asked me about a report that Moderna’s monkey trial of an Omicron-specific booster wasn’t any better than the existing vaccine. What should we think about that?

Omicron-specific boosters!

Everybody’s worried about Omicron, in spite of the fact that it’s almost over. Omicron is so infectious that it burns through a population pretty quickly.

Since, apparently, here at Chez Weekend we’re all about sewage lately, have a look at the viral RNA levels detected by Boston’s MWRA as of today. See that big peak at the right, almost back down to background? That’s the Omicron wave. Cases, hospitalizations, and deaths will still be peaking, but the Omicron viral wave is coming to a close.

So it’s with a sad sense of irony that we contemplate Omicron-specific boosters, as a classic case of too little, too late. Though to be fair, given Omicron emerged last November, nothing could be fast enough. Don’t blame the vaccine companies for that.

Still… it makes sense to use the nimble mRNA technology to update vaccines. Even last March, when the Big Bad du jour was the Beta variant (remember Beta?), both Moderna and Pfizer were working on variant-specific vaccines, and the FDA had pledged expedited review akin to an annual vaccine for flu variants.

So it’s only natural to expect work on an Omicron vaccine. If it were available now, I’d certainly take it!

Let’s check in on the progress of that work.

Some claims about the Moderna Monkey Trial for Omicron

(I can’t resist the slightly improper phrase “Moderna Monkey Trial”, because it evokes the “Scopes Monkey Trial” when American fundamentalists got whupped. Maybe we need to do that again, to suppress the right-wing hysterical irrationality here in the US?)

My correspondent asked about an article in Ars Technica ^[1] claiming that the result of the Moderna Monkey Trial showed vaccine efficacy in monkeys ‘only as good as’ the current vaccines. Now, 2 things immediately stood out to me:

1. It frosts my cheese when someone sneeringly dismisses something as ‘only as good as’ the most effective vaccines in human history, developed in record time. That’s like saying you’re only a so-so skier, because you’re ‘only as good as’ a gold medalist. That phrase ‘only as good as’ is doing a lot of lifting here, and not in a good way.
2. Animal trials are not about showing efficacy in the first place! They’re about safety (“did any of the monkeys catch fire and explode?”) and about PK and PD (“did the drug levels in the blood go down about like you thought?”, and “did the drug get into the right compartment, at the right time, at the right level, and for the right duration?”), and about dose-finding (“do you know how much is enough?” and “do you know how much is too much?”). Sure, if you see efficacy, it’s fine to report that. But it’s not the main thing.

Some digging is required here, to route around the damage caused by bad science journalism. The actual scientific report is still a preprint ^[2], but it’s available to read. Sure enough, it only uses the word “efficacy” in the introduction and in justifying the use of a primate model. It’s all about memory B cells, T cells, and the like. The conclusion is more or less, “No monkeys currently on fire. All monkeys show immune stimulation. Sure looks like a vaccine.” And that’s what you should expect!

So… one reasonable reaction would be: “move along kid, nothin’ to see here.”

But that’s boring. Let’s examine the reasonableness of making any claim about efficacy based on the Moderna Monkey Trial. (Sheesh, I love that phrase.)

What efficacy means for $N = 8$

Our intrepid journalist at Ars Technica cites herself as a source for efficacy of the current vaccines against Omicron. ^[3] She claims Omicron efficacy against the need for urgent care of 82%, and efficacy against hospitalization of 94%. That seems to be fine, sourced from 2 MMWR’s from the CDC. ^{[4] [5]}

So: can we conclude anything statistically significant about vaccine efficacy in comparison to those values of 82% or 94%? After all, finding a result better than 94% would probably require a lot of samples, since there’s not much room up at the top there!

Did we have a lot of samples in this study? No. We had $N = 8$.

Upon seeing that, we’d be within our rights to throw up our hands in disgust and go see what’s on TV. That would be about as productive as anything else. But… here at Chez Weekend, we like to dig. So let’s dig.

There were 4 treatment animals who got 2 ordinary Moderna shots plus the Omicron booster, and 4 control animals who got 3 ordinary Moderna shots. That means there are only 25 possible outcomes, if you’re measuring who got COVID-19 and who did not: 0-4 in the control arm, and 0-4 in the treatment arm. (NB: This is not what the paper is about! The paper is about measuring B cells and T cells and lions & tigers & bears, oh my in both arms. They concluded the results were comparable, as you would expect, and thus could proceed to human trials.)

So we wrote a little R script to check what the observed efficacies and their 95% confidence limits would be in those 25 cases. ^[6] The confidence intervals are computed with a simple binomial confidence interval, not my fancy-pants beta ratio method, which I still have not finished. Feel free to berate me about that.)

Recall that for $N_c$ controls and $N_t$ treatment animals, of whom $K_c$ and $K_t$, respectively, get COVID-19, the point estimate of efficacy is defined as:

\[\begin{align*} \mbox{Efficacy} &= 1 - \frac{\Pr(\mbox{COVID-19} | \mbox{treatment})}{\Pr(\mbox{COVID-19} | \mbox{control})} \\ &= 1 - \frac{K_t / N_t}{ K_c / N_c} \end{align*}\]

Here’s what the 25 possible outcomes look like:

• All efficacy computations are for $N_c = 4$ controls and $N_t = 4$ treatment animals.
• The rows are $K_c$ values, the number of sick animals in the control group; the columns are $K_t$ values, the number of sick animals in the treatment group.
• The values in each cell are the efficacy point estimate from the equation above and the 95% confidence limits based on a scaled binomial confidence interval.
• The first row is peculiar but sensible: it reflects $K_c = 0$, no sick controls.
  • The upper left corner is sensibly NaN because it’s computing 0/0, i.e., what happens if you run a COVID-19 trial and there’s no COVID-19: you get no information!
  • The other elements of the first row are sensibly $-\infty$, because $K_c = 0$ means dividing by 0, i.e., given the nonzero cases in the treatment arm, the treatment arm was infinitely more risky than the control arm.

A few things should stand out:

• In no case was it possible to say the 95% lower confidence limit was greater than 80% or 94%, the efficacy of the existing vaccine.
• In fact, only in the 3 outcomes in the lower left, in bold and red, would it have been possible to say with 95% confidence that efficacy was greater than 0%!

Summary: With an $N = 4 + 4$ trial, it is simply not mathematically possible to show that efficacy was statistically significantly better than the existing vaccines. To complain otherwise is a misleading canard.

The Weekend Conclusion

Sure, the trial didn’t prove the omicron booster was better. It could not possibly have done so, and very sensibly was not designed to do that. Thinking that this constitutes failure is just ignorant.

The trial did succeed at doing the sensible thing: demonstrating safety in primates, and demonstrating immune reaction in the all-important memory B cells and T cells that we want to see. A result like this should, and probably will, clear the way for a human trial.

Good job on this experiment. Somewhat less good job on the science journalism. (But, to be fair: it was good journalism to cite the primary sources so we could check up on the conclusions.)

---

Notes & References

1: B Mole, “Moderna’s omicron booster was only as good as current vaccine in monkey study”, Ars Technica, 2022-Feb-07. ↩

2: M Gagne, et al., “mRNA-1273 or mRNA-Omicron boost in vaccinated macaques elicits comparable B cell expansion, neutralizing antibodies and protection against Omicron”, bioRχiv, posted 2022-Feb-02. DOI: 10.1101/2022.02.03.479037.↩

3: B Mole, “Unvaccinated 5X more likely to get omicron than those boosted, CDC reports”, Ars Technica, 2022-Jan-21. ↩

4: AG Johnson, et al., “COVID-19 Incidence and Death Rates Among Unvaccinated and Fully Vaccinated Adults with and Without Booster Doses During Periods of Delta and Omicron Variant Emergence — 25 U.S. Jurisdictions, April 4–December 25, 2021”, US CDC Mortality and Morbidity Weekly Report 71:4, 132-138, 2022-Jan-28. ↩

5: MG Thompson, “Effectiveness of a Third Dose of mRNA Vaccines Against COVID-19–Associated Emergency Department and Urgent Care Encounters and Hospitalizations Among Adults During Periods of Delta and Omicron Variant Predominance — VISION Network, 10 States, August 2021–January 2022”, US CDC Mortality and Morbidity Weekly Report 71:4, 139-145, 2022-Jan-28.↩

6: Weekend Editor, “R script for vaccine efficacy in Moderna Monkey Trial”, Some Weekend Reading, 2022-Feb-09. ↩