What's in the Phase 1/2 readout of the Pfizer/BioNTech COVID vaccine?

Somebody asked me what’s in the Phase 1/2 readout of the Pfizer/BioNTech COVID vaccine. Of course they wanted to know about the trial results, but they also wanted to know if it is an investment signal. It is not.

It all started with some kind of news media article. A bit of advice: please don’t bother with the news media on the subject of science. They’re so captivated by narrative that they’re usually mathematically illiterate, and generally worthless on scientific subjects. I went to Statnews, which carries stories about biotech/pharma/medicine with at least a clue about statistics. Sometimes not much more than a clue, but at least a clue.

Most importantly, you can ignore the Statnews article as well, because they usefully link to the primary literature which is where you get the real goods. In this case, it’s a link to the paper on MedRXiv. The nitty-gritty details are in the study’s page at ClinicalTrials.gov: NCT04368728. I’m just going to take a quick look through the paper for our purposes here, not at the ClinicalTrials.gov page.

Cautions & Encouragements

Caution #1: MedRXiv, in imitatione of the physics server arxiv.org, is a preprint server. That means it contains papers that have been written up and are being proposed for publication and the required peer review. They have not yet been peer reviewed, and thus are suspect from the very start. I’ve never published a paper that didn’t have to be revised because of peer review, so keep that in mind. Everything will get toned down in the final, accepted version – if it even makes it that far.

I looked over the authors, and didn’t see anybody I know. That’s fine; there are an awful lot of people working in this area and I have only a few friends & acquaintances doing that. Slightly better news: I didn’t see anybody I knew to be a dangerous idiot on the author list, either. (And I do have a list of dangerous people to warn about when I see them, and have done so before in my offical capacity as a researcher. These guys seem pretty clean, as far as I can tell. So good for them.)

Encouraging item: Ok, Pfizer is pretty big, and will be able to manufacture a lot of doses if they get approval (not as much as Sanofi which has a long vaccine history, maybe not as much as Lonza + Moderna; but still a lot). Pfizer haven’t always the best scientific reputation on the street, and occasionally I meet scientists from Pfizer who complain about a few management dullards that didn’t understand science, or even understand that they should understand science. (On the other hand, let’s be fair: almost every good scientist I know thinks corporate upper management don’t quite get science. So there’s that. Confirmation bias^[1] is a thing.) But… they’re working with BioNTech, who are quite good. They specialize in immuno therapies. I’ve worked (indirectly) with BioNTech in oncology, with quite satisfactory results. So the combination of Pfizer’s muscle and BioNTech’s smarts might be a pretty good mix, if they can keep management fingers out of the gears. Which is probably likely to succeed, given the intense scrutiny surrounding COVID vaccines.

Caution #2: Right from the start, we see in the title that this is a Phase 1/2 study and the subjects were age 18-55. Now, there’s nothing wrong with that; that’s the way it should be. But it means: don’t jump the gun and assume this is a vaccine!

1. Phase 1 (and don’t get me started about why there even is a thing called Phase 1/2) is all about safety: give your therapy to a small number of people, and see if there are any adverse events. (“Adverse event” usually means very bad; “severe adverse events” means life-threatening.) So we’re not really even interested here in efficacy (antibody production, infection resistance upon challenge), we’re just interested in safety. We should treat anything they say about efficacy very skeptically, i.e., “that’s nice, but what adverse effects did you see?”

2. The subjects were all young to mid-life (18-55) and healthy. That’s fine for Phase 1/2; again, it doesn’t look like anything is amiss there. But: this is not the highest-risk population! The highest risk populations are the elderly (not 18-55) and those with any number of pre-existing conditions (obesity, diabetes, asthma, …). That population, with the highest unmet medical need, is also the hardest population in which to induce an immune response. There’s a reason the flu vaccine sometimes comes in a special high-dose formulation for elderly people. So what they did here was test a healthy population not most at risk. That’s sensible: for a safety trial, you want a test population who are least likely to be endangered.

So the top-line thought I have is that this is all very nice, but it is (a) a pre-publication, non-peer-reviewed report that is subject to significant revision or even retraction, (b) it is a safety trial whose efficacy results are a secondary objective, and (c) it tests a population who, while not medically irrelevant, are very much not the high-risk population.

Encouraging item: I like that it was both placebo-controlled, and double-blinded. Especially in emergencies, people try to skirt around this and end up crapping all over their own data. As I said above, this looks like a well-designed, responsibly run, early clinical trial for safety. Well done, Pfizer & BioNTech; but nobody should interpret this as actionable investment news.

The vaccine itself is an mRNA vaccine. They use (probably) a bioreactor platform to produce incomplete viral mRNA, i.e., some of the viral genetic information but not enough to assemble the virus itself, for safety. This is delivered in a lipid nanoparticle capsule, which survives in the blood just long enough to get taken apart in the liver. The liver, among other tissues, will then recognize the viral mRNA, and concludes Nothing Good Will Come of This, and calls in the immune system cops. The idea is that the mRNA is taken up by your cells, your cells start to manufacture viral proteins (not enough kinds to make all the parts of the virus!), and then they look at the proteins & think, “Wait, what? That doesn’t belong here!”

They sensibly chose the spike glycoprotein of SARS-CoV-2, which makes the most sense immunologically. It all smells pretty good, just from a study design point of view, for an early safety study.

Again, this is fine. Everybody wants to try an mRNA vaccine, because it’s fast. Moderna has the first one (“mode RNA”, get it?), which they were pulling out of their bioreactors within, I think two weeks of the publication of the viral genome by the Chinese authorities. (Really, strikingly, well done.) But ok, Pfizer/BioNTech has one, Sanofi has one, etc. If these mRNA vaccines work, it’ll be bigger news than just for COVID-19 because we might be able to make a lot of different vaccines faster.

Caution #3: Despite how attractive the mRNA mode is for vaccine development, and despite years of trying, and despite the founding of companies like Moderna on this premise, no mRNA vaccine has ever made it to approval. (Yet.) Just sit with that thought for a moment: despite the potential to make billions if it works, and lots of smart, well-funded people trying, we’ve never yet succeeded. Maybe COVID-19 will be the first time, maybe not. It’s a bet, and it’s probably long(ish) odds against it.

This was a dose-escalation design, i.e., they first did a group at low dose, and if nobody died they recruited more subjects at the next higher dose (see p. 6 for a figure illustrating this). All very sensible. But… they screened a bunch of subjects, and then only vaccinated 12 with the real vaccine + 3 with a placebo in each of 3 dose arms. That is, we’re looking at a 12 vs 3 comparison in each dose group. This is fine for a safety study, and this is a safety study so all is well. But…

Caution #4: While adequately statistically powered for a safety study (I think; I haven’t done either a frequentist or Bayesian power calculation), it is almost certainly woefully underpowered for drawing conclusions about efficacy. That’s not a bad thing, since these people are sensibly doing a safety study. It’s a warning to investors: don’t be stupid. This is not a sign that Pfizer will make bajillions.

Caution #5: Everybody got a variety of side-effects, and at first glance, the severity appears to be dose-related, i.e., pointing the finger at the treatment as cause. How severe? Well, there’s a whole nomenclature for this sort of thing, but generally “serious” to a clinical trial means “would lay you out flat on your back and make you demand to go to the hospital”. I.e., you do not want to experience that. There is an even higher level, grade 4, not observed here (fortunately), which involves things like necrotic tissue (“bits of you are dying”). See especially Figures 2&3, pages 7-8.

Encouraging item: Most everybody got some degree of immune reaction, compared to the negative controls. Most everybody only got it after the second dose, though; that’s not necessarily bad. See Figure 4, pp 9-10. Rather gratifyingly, they used convalescent sera as a comparison, i.e., ensuring that the degree of immune reaction to the vaccine was comparable to people who had actually recovered from COVID-19.

Caution #6: To their credit, they mention some cautions on page 5:

• Nobody knows how much immunogenicity is enough to confer even partial immunity.
• They only looked 2 weeks past the 2nd dose, so they have no idea how long-lasting this is. (Though they promise to try to track their subjects for 2 years. That’s still a loooong time in the future.)
• They also admit what I pointed out above (before I’d read the rest of the paper, or I’d have said it more nicely) that the population of healthy 18-55 year olds is not the medically most relevant. Bravo to them for honesty. I’m slightly impressed at the integrity here; I was prepared to take a lot more sideswipes at them, but they really are playing it straight even if the media will garble it all.
• They also claim to be studying a second cohort of age 65+, and “will report” later. Good on them to have done that second, after the safety trial in a younger, more robust population. And they’re enrolling a population with pre-existing conditions later.

Ok, that’s as much as I can get after spending about an hour with the paper (not scrutinizing everything in it by any means!), and not looking at the ClinicalTrials.gov page at all.

Summary

This looks like a good, credible, well-designed, early safety study. But the cautions above are now something I believe are even more important: it’s not yet peer-reviewed, it’s too early to have any idea whether it will work, no mRNA vaccine has ever before made it to approval, everybody got some degree of relatively nasty side-effects, and the population is not the most relevant one. But, as hopeful signs: the side-effects were tolerable (clinical speak for “horrible, but survivable without hospitalization”) and there was induced immune reaction (though it matched convalescent patient sera, nobody knows if that’s enough, and at 2 weeks we have absolutely no idea if it’s long-lasting).

So… well done, Pfizer + BioNTech. If I were peer reviewing this – well, after a lot more work than I’ve done here, since I tend to be a pretty careful reviewer – then I’d probably give it a thumbs up for publication.

Investing

Investing, though?

Look, people bidding up Pfizer’s stock over this are idiots. They have literally no idea of the content of the paper we just went through, and how unlikely it is ever to affect Pfizer’s revenues. For example: in 2019, almost all of Pfizer’s vaccine earnings were from a single vaccine (for pneumonia), and that was a tiny fraction of their total revenues (looks like about 10% of total revenues?). People putting down money on this are taking a very long shot, and doing it for something that might affect a fraction of the 10% of Pfizer revenue from vaccines.

Also, nobody’s going to make a bundle on this. They’re all going to have to give it away, pretty much. Imagine what would happen to a pharma company that wanted to charge \$2k/person for a vaccine, in a world with 8 billion people many of whom are poor? They’d lose their corporate charter more or less immediately, or face nationalization. (BTW, Gilead is charging \$3k/patient for remdesivir, which only shortens the hospitalization of the most severe patients by 4 days, and does not affect death rates at all. I wonder if Gilead will survive the storm that is sure to cause? Or if governments will invoke the “compulsory licensing” provisions in law to just take it away? Vaccine makers are, trust me, not unaware of this.)

---

Notes

1: Hey, I believe in confirmation bias. If I keep seeing examples of confirmation bias everywhere, is that an example of confirmation bias about confirmation bias?! ↩