Analysis: how Oxford vaccine trial results raise more questions than answers

Good news fades fast. On Monday Oxford University and AstraZeneca announced its vaccine was “up to” 90 per cent efficacious against Covid-19. But by yesterday US experts were picking holes in the results, with some saying there was little chance of it gaining quick regulatory approval in the US without further data.

Reading the runes, AstraZeneca yesterday said it would conduct an additional trial to assess how different dosing impacts the efficacy of its Covid-19 vaccine.

Regulatory issues are unlikely to be unique to the Oxford AstraZeneca jab. Licencing vaccines is a complex business and those responsible will not be bounced by press release - the device used by all three companies to publicly disseminate their data.

Criticism of the Oxford trial falls into three main categories: a lack of transparency, imperfect controls and over-interpretation of the small amounts of data collected to date.

On transparency, Oxford finds itself at a disadvantage to Moderna and Pfizer which published the trial protocols for their vaccines in advance. Oxford, by contrast, has published the protocol for only one of its phase three trials and did so only in October.

Oxford “get a poor grade for transparency and rigor when it comes to the vaccine trial results they have reported”, said Natalie Dean, a biostatistician and an expert in vaccine trial design at the University of Florida.

To a British audience used to medical paternalism this might not seem important, but it’s a serious matter in the US which has a long history of medical scandals centered on trial secrecy, several based on race.

Oxford is also taking flak for the way it finessed its results in Monday’s press release. It stated one dosing regimen of the trial had efficacy of 62 per cent and another 90 per cent. The average, it said, was 70 per cent.

This, say experts, is too simple an interpretation. The 90 per cent data point relates to part of the trial where the age profile of participants was younger (under the age of 55) and the total number of participants was three times smaller than in the poorer performing group (2,700 compared to 8,900).

In short, the difference in outcome may have nothing to do with the dosing. Instead it may relate to the age of participants, or may just be a statistical quirk.

The original difference in dosing was down to a manufacturers' error and in a statement Oxford said it cleared this with the regulator once they realised the mistake.

A third criticism relates to Oxford’s control groups. The Oxford team has merged data from two separate trials, one in Brazil, one in the UK. Unfortunately, the protocols for both differ and a different placebo was used in each.

According to Hilda Bastian, an expert in assessing health claims, the fact that researchers have had to combine data from two different trials is a “red flag”.

“These two studies were substantially different from one another: they didn’t have standardised dosing schemes across the trials, for one thing, nor did they provide the same ‘control’ injections to volunteers who were not getting the experimental Covid vaccine,” she wrote in Wired magazine.

It would be wrong to make too much of these criticisms or assume similar problems will not emerge with the Moderna and Pfizer results.  They are only to be expected given the circumstances of the pandemic and crunched timelines involved.

While all three trials have involved tens of thousands of volunteers, only half have received a vaccine and the numbers contacting Covid stand at only about 100 in each.

Such small numbers can provide confidence of efficacy at a high level but they cannot provide statistical certainty when looking at subgroups.

The chances of flipping a coin and it, by fluke, landing on heads 100 times in a row is about one in 500 billion. But if you take a subgroup of 10 of those flips - those performed by people over 75, for example - the probability falls to about one in a thousand.

Regulators will not only be looking for signals of efficacy but safety. And here too the numbers from each of the Covid trials are small, especially for the Moderna and Pfizer vaccines which rely on an mRNA technology which has never been used before.

Geoffrey Porges, a biotech analyst at SVB Leerink, told Bloomberg News: “If it's a one in 10,000 adverse event you might only see one in the case of these 11,000 vaccine subjects and it’s easy to dismiss. But when you get to 30,000 at least if you have two or three you might start to see the signal. And those signals become very large signals when you go out and vaccinate 10 million people.”

What does all this mean going forward? Should we as potential consumers be worried?

Probably not but we need to see the licencing of these vaccines for what it is: emergency use clearance in a pandemic which is killing hundreds of thousands and closing down economies around the world.

Regulators, and ultimately the politicians, will not in the end be making a perfect call. They will be weighing the data available and balancing risk against reward.

Read more:  The elite concierge that will get his clients first dibs on the Covid vaccine

Protect yourself and your family by learning more about Global Health Security