Effectiveness reviews

In recent years HIV prevention workers, researchers and funders have become increasingly concerned with identifying how best to spend limited funds. This requires a better understanding of which HIV prevention initiatives have been most effective, and which ones are least effective.

A study funded by the UK Health Education Authority (now the Health Development Agency) attempted to identify examples of effective interventions, using criteria developed in other health care fields (Oakley). These criteria demanded that studies reporting on the effectiveness of prevention interventions should ideally display the following characteristics:

• A clear definition of aims.
• A description of the intervention package and design sufficiently detailed to allow replication.
• A randomly allocated control group or matched comparison group.*
• Data on numbers of participants recruited to the experimental and control groups.
• Baseline (pre–intervention) data for both groups.*
• Post–intervention data for both groups.*
• Drop–out rates for both groups.
• Findings for each outcome measure as defined in the aims of the study.*

However, the review used only four of these characteristics (those marked with an asterisk) when defining studies with a `sound' methodology. Paradoxically, for a review intended to develop awareness of effective interventions, replicability (point 2) wasn't amongst these criteria!

The study found that just 18 out of 68 outcome evaluations could be described as methodologically `sound'. Just nine of these studies were conducted with adults in high-risk groups. These studies are amongst those discussed in more detail in What is known about the effectiveness of interventions? later in this chapter. A number of other studies published since this article appeared are also discussed below.

Another review discusses the question of sound methodology in more detailed terms. A review funded by the US National Institutes of Mental Health and the US Centers for Disease Control proposes a series of standard reporting requirements in HIV and STD prevention behavioural interventions which will allow comparisons between effectiveness studies.

• Describe recruitment and sampling methods: In order for any study to be replicated it is essential that the description of the methodology is as full as possible. It is also important for the reader to be able to judge whether any aspects of the methodology could have biased the results. For example, was the sample a volunteer group or random sampling? How were they selected?
• What proportion of potential participants were recruited into the study?
• What was the refusal rate? What differences exist between participants and non–participants?
• Define the sample size required to produce statistical significance. When a study is designed, statistical methods should be used to define how large a sample will be required in order to detect a certain magnitude of effect on a behavioural or biological outcome. For example, researchers might predict that in order to detect a 20% reduction in syringe sharing, they would need to recruit 100 people into a study in order to be certain that a 20% reduction wasn't due to chance, but to the effects of the intervention. This is called statistical power.
• Was there a control group, and if so, how were people randomised? A control group can guard against the effects of being studied (such as people telling researchers what they think they want to hear).
• Report length of follow–up. This allows readers to judge whether there is evidence of a sustained effect.
• Describe the theory of behaviour change and education underlying the intervention. This allows the reader to judge whether the design of the intervention is likely to test the theoretical assumptions underlying the intervention.
• Describe the intervention. Too many study reports fail to describe the intervention in sufficient detail. What were the messages communicated to participants? How many sessions/exposures were there?
• Describe outcome measurements used. A more detailed discussion of outcome measures follows later in this chapter.
• Report results of mediation analysis. Mediators are the elements of an intervention which mediate between the intervention itself and the behaviour change. “Mediators are generally conceptual variables designed to be addressed by the intervention because they are believed to cause behaviour change, such as perceived social norms, skills, self–efficacy or perceptions of risk. Mediation analysis involves demonstrating that the intervention changed the mediator as well as the behavioural outcome” and that controlling the mediator also affects levels of risk behaviour. If this relationship cannot be proven, then another factor is responsible for behaviour change, and the theoretical relationship between the mediator and the behaviour targeted for intervention is unproven.

References

Oakley A et al. Behavioural interventions for HIV/AIDS prevention, AIDS 9: 479–486, 1995.

O'Leary A, DiClemente R et al. Reflections on the design and reporting of STD/HIV behavioural intervention research, AIDS Education and Research, 9 Supp A, 1997.