The modest protective effect of the HIV vaccine combination used in the Thai vaccine trial announced last month is statistically significant and not the result of massaging the statistics to produce a positive result, study investigators said at the AIDS Vaccine 2009 conference in Paris today.
The protective effect of the vaccine had been called into question following the release of headline results from the study in late September, which showed that the vaccine reduced the risk of HIV infection by 31%, a result that was statistically significant.
However, at the time of the announcement attention was drawn to the very wide confidence interval of this estimate. Statisticians estimated that the result lay between 1.1% and 51.6% with 95% confidence; in other words, there was a 5% possibility that the result lay outside these bounds.
Subsequently, news leaked on other data from the trial, which suggested that if analysed in other ways, the results were not statistically significant. In particular, concerns were raised about why the results of the less encouraging per-protocol analysis were not released to the world’s press at the same time as the headline result.
Today Dr Nelson Michael of the US Military HIV Research Program explained the three analyses of the study and why the investigators chose to release one particular analysis last month, ahead of the AIDS Vaccine 2009 conference.
Three analyses of the study were planned:
- An intent-to-treat analysis, which included all trial participants randomised, regardless of whether they received the vaccine or not.
- A modified intent-to-treat analysis, which excluded any randomised participant who was discovered to have evidence of HIV infection through viral load testing prior to vaccination. This analysis includes study participants who may have missed some of the vaccinations, and goes some way to estimating the real-life efficacy of the vaccine.
- A per-protocol analysis which excluded all participants who missed any doses of the vaccine, or who received vaccinations on days other than scheduled study visits. A per-protocol analysis in a vaccine study is designed to test the efficacy of a particular vaccination schedule, and is most useful in a study that is intended to lead to product licensing.
The headline results of the study, released last month, indicated that the vaccine reduced the risk of infection by 31%, but the statistical confidence intervals were very wide (between 1.1% and 51%). Nevertheless the result was statistically significant (p=0.04).
Other results were not released to the press in September, said Dr Jerome Kim of the US Military HIV Research Program, because they were already embargoed prior to the AIDS Vaccine 2009 conference. But researchers had to honour their commitment, made at the outset of the study, that the Thai people would be the first to hear the result of the trial, so the modified intent-to-treat analysis was revealed in September.
The strict intent-to-treat analysis, however, showed that the reduction in the risk of infection in the vaccine group was not statistically significant (-26.4%, 95% confidence interval -4% to 47.9% (p=0.08). This analysis included seven participants who turned out to have been infected at the time of randomisation, prior to the first shot of vaccine. These infections were identified when stored samples taken at baseline were tested after the participants tested positive at the time of their fourth vaccination.
Similarly, the per-protocol analysis, which excluded around 25% of study participants and thus failed to capture around 31% of infections that occurred during the study (just over half of them in the first six months), showed a reduction in risk of 26.2% (95% confidence interval -13.3% to 51.9%, p=0.16).
The full results were also released online today by the New England Journal of Medicine and, in their report, the investigators note that the result of the modified intent-to-treat analysis remained statistically significant regardless of the statistical method used to test significance. The investigators used no fewer than six different tests to query the magnitude and robustness of the effect observed, and found that all produced results within the same range, a p value lying somewhere between 0.03 and 0.05.
The researchers concluded that the modified intent-to-treat analysis, which had always been planned as part of the study and which was used throughout the study as the primary analyses to determine if the trial should be stopped on the grounds of futility, gave the most clinically useful information for making future decisions about how to take the vaccine forward, since it most closely reproduced the likely conditions in which a vaccine’s effectiveness would be judged in the field.