Skip Traditional Phase 3 Trials for High-Burden Vaccines

Deploy After Phase 2, Test During Rollout (Day 4 Inkhaven post)

Epistemic status: Exploratory idea. Promising but highly uncertain. Drafted in one day. Please don’t make irreversible decisions based on this, but also I continue to think it’s worth investigating. Will update if I have clarity.

Claim: Vaccines for novel pandemics and significant endemic diseases should skip Phase 3 Trials.

Instead, we should do an experimental roll out of vaccines after Phase 2 and conduct rolling post-deployment trials, with the option of recall.

This saves several months to 4 years, and between a few hundred and hundreds of thousands of lives per vaccine hastened. I expect minimal effects on vaccine safety harms, and minimal loss in public confidence and perception, though more research is needed¹.

Please comment here or email me if you’re interested in collaborating or investigating further.

Intro: What are Phase 3 trials?

After virtual, lab, and animal studies, clinical trials for new medications typically go through 3 phases:

1. Phase 1: Experimental study of <100 usually healthy participants to determine glaringly obvious safety issues and initial side effects

2. Phase 2: Larger study of a few hundred to few thousand people, to test initial signs of drug/vaccine efficacy² and further track side effects.

3. Phase 3: The largest rollout, to test thousands. This can usually take 1-4 years. COVID vaccine Phase 3 trials were unusually³ short (on the order of 3-6 months).

   1. For vaccines specifically, Phase 3 trial sizes can be very large (about 30,000 people).

   2. This is because unlike with other drugs, where drug trial volunteers already have the disease, vaccines are tested preventatively, on people who don’t have a disease. So since most people in neither the treatment nor the control group will be infected, it’s harder to tell vaccine effectiveness unless you have tens of thousands of trial participants.

Source: From: https://www.universityofcalifornia.edu/news/new-phase-clinical-trials

The Problem

The problem is that Phase 3 trials for vaccines don’t actually offer much benefit: 85% of vaccines that make it past Phase 2 vaccine trials will make it past Phase 3.

[...]

Source: Table 2 ofhttps://academic.oup.com/biostatistics/article/20/2/273/4817524?login=false

This means, the vast majority of the time Phase 3 trials are conducted, it’s an “extra safety check”, not actually a critical experiment for vaccine effectiveness.

Further, most Phase 3 trials that fail do so for reasons of drug efficacy or organizational effectiveness, not safety. Only 17% of Phase 3 trials fail for reasons of safety.

So only ~17% * (100%-85%) ~= 2.5% of all vaccine Phase 3 trials end in people discovering the vaccines are unsafe.

In the meantime, Phase 3 Trials delay vaccine rollout by 1-4 years, costing thousands of lives for some diseases and hundreds of thousands of lives for the worst diseases.

The Solution

I believe we should skip the current Phase 3 trial regime for vaccines, at least for novel pandemics and for “serious” endemic infectious diseases with very large human burdens (TB, malaria, etc). Instead, after Phase 2, we should immediately initiate an “Experimental vaccine rollout”⁴ concurrent with post-deployment trials.⁵

Either

1. Marginal people can sign up for trial vs control group, OR

2. People who would otherwise get the vaccine can volunteer (possibly with stipends) for being in an experiment group with 50% chance of getting a control.

Either way, we can run a full RCT with the same or higher⁶ level of rigor of a normal Phase 3 trial, without the debilitating human cost of waiting for the results of a Phase 3 trial before rollout.

Since dose production and distribution will be limited in the beginning, running the trials should not limit the doses distributed significantly.

The Expected Benefit

It’s hard to estimate the expected benefit per vaccine, as it is heterogeneous across diseases, candidate vaccines, and production curves.

But consider a hypothetical disease, vaccine pair where:

• The disease kills 100,000 children annually

• with a candidate vaccine that passed Phase 2

• and has a 85% chance of 50% efficacy.

If we can rush the vaccine deployment by 2 years, we’d thus avert 100,000 deaths/year * 2 * .85 * 0.5 = 85,000 deaths. That’s 85,000 lives saved, or millions of DALYs averted!

More realistically, if in the first two years we are ramping up enough production to only cover about 10% of the affected population by the end, that is still on the order of 4,000 lives saved!

(In comparison, the expected cost from rolling out Phase 2-passing vaccines with significant adverse effects is approximately negligible, unless I’m dearly miscalibrated about typical negative side effects of vaccines that already passed trials with thousands of participants)

As a practical example, malaria kills about half a million people per year, mostly children. Meanwhile, RTS,S (Mosquirix) took decades to get through trials (including multiple years for phase 3 trials starting in 2009), finally approved in 2021. Even at modest efficacy (~30-40%), earlier deployment would have saved hundreds of thousands of lives.

The counterfactual deaths during the Phase 3 delay are thus enormous. Averting such deaths is critical.

Another consideration is novel pandemics. Of course, COVID (which killed over 20 million people and caused tens of trillions of dollars in economic damages worldwide) was bad. But there’s no law of nature that says COVID is the worst novel disease we’d face this century, or anywhere close.

If we ever encounter another pandemic many times worse than COVID, I dearly hope we’d already have the policies in place to rapidly produce, distribute, and test the best known vaccines of the time.

Reasons I Could Be Wrong/Future Work

1. The “85% of vaccines that passed Phase 2 passed Phase 3” figure was from sampling vaccine trials in 2000-2015. There’s some initial evidence that the numbers have since gone down, though I’d be curious whether the trend continued.

   1. That said, on purely life-saving grounds, the important question isn’t “what % of vaccines that pass Phase 2 pass Phase 3” but comparing the costs of premature rollouts (safety incidents) vs delays (how many people die from avoidable causes due to slower vaccine rollouts).

2. I’m currently assuming that the safety issues uncovered in the 17% of phase 3 trial failures aren’t actually as bad as getting the worst endemic or pandemic diseases, but I didn’t do a dive to be sure. Would love to read more/see more details!

3. Even if my policy ideas are sound on a public health level, maybe they’re bad on a public relations level. E.g., maybe experimental vaccine rollouts for less tested vaccines, or expensive, graphic, and public safety failures will sharply increase vaccine hesitancy

   1. Fortunately, public relations effects are not Mysterious and Unknowable. They can be scientifically studied and tested, just like any other scientific and social science question

4. I don’t have a good mental model of how policies get changed. I assume it starts with conversations and idea blog posts like this one evolving into more formal investigations (white papers, published academic papers) by established sources, maybe more public conversations until elite “consensus” (or elite consensus in specific subfields) changes and policymakers agree it’s kind of reasonable or even inevitable before it subsequently becomes policy.

   1. But I could be way off, and besides, I’d love to know of a faster way! Every life is precious, and every life lost due to our inactions a tragedy.

Get in Touch

If you are interested in advancing this idea further, and especially if you have experience, expertise, or authority in related matters (e.g. if you’re an academic or policymaker in a related area, or know people who are), please consider contacting me!!! My Substack DMs are always open.

I’d be very keen to brainstorm together on ways to reduce our uncertainties. I’d love to advance this idea further, and make real progress on such an important issue with a real chance of improvement in human lives.

I’m especially interested in people with connections to decision-makers outside the United States (e.g. WHO, African CDC, British NHS), as I’m pessimistic that the US can improve on vaccine matters in the next 3 years.

Of course, if you think contacting me does not advance our mutual interests (for either rigor or political reasons), and you’d rather progress this idea without connecting, I totally understand. Saving lives is so much more important than my ego, and I absolutely would not begrudge anybody’s pragmatic choices here.

1. The language in this post is framed with significantly more certainty than I all-things-considered think is warranted. However, (unlike some of my other posts) there are many roadblocks before my vision becomes reality, and approximately zero chance that my post becomes policy without further checks. I’m keen to discuss further and help design further research protocols and experiments to advance or disprove these arguments.

2. ie, whether the drug actually works

3. One of the forecasts I was proud of: in mid-March 2020, I assigned a high probability of successful Phase 3 vaccine trials for COVID before the end of the year:

4. Explicitly tagged as such

5. we can call it Phase 4, Rolling Phase 3, Emergency Phase 3, or some other fancy term people like.

6. Recruiting for Phase 3 clinical trials is a sore point for many vaccine manufacturers. Changing the script from “volunteering to possible receive a vaccine” to “possibly receive a control” may allow for faster trial participant recruitment. Further, the trial testing conditions might be more ecologically valid if it’s part of a larger rollout than in a “normal” Phase 3 trial.