Outcome measures

Researchers and prevention workers have attempted to assess the effectiveness of interventions by using a number of measures:

• Changes in knowledge and attitudes.
• Numbers of persons reached.
• Changes in uptake of condoms or injecting equipment.
• Changes in reported behaviour amongst a cohort.
• Changes in reports of sexually transmitted infections.
• Changes in HIV incidence or prevalence in the population.
• Changes in HIV incidence or prevalence in a cohort.

This section looks at the value of these measurements, with reference to recent examples from international research, and then goes on to summarise what is known about effective interventions. This list of measurements covers those used in process evaluations and those used in outcome evaluations.

Knowledge and attitudes

Investigation of the effects of HIV prevention campaigns on the knowledge and attitudes of the target population was one of the first measurements of effectiveness to be adopted by researchers. It was considered to be a crucial first step in determining whether or not information about HIV and AIDS had been received and understood by the target population.

During the 1980s knowledge and attitudes surveys carried out in the UK showed wide and rapid dissemination of messages about HIV infection and AIDS, but also showed alarming levels of miscomprehension of these basic messages. These surveys tended to investigate the impact of basic HIV awareness campaigns, and provided useful baseline data.

Such surveys are of limited usefulness today except in situations where new concepts are being introduced to a population. For example, knowledge and attitudes surveys amongst gay and bisexual men in the UK over the past eight to ten years have repeatedly demonstrated high and unvarying levels of knowledge about AIDS, HIV, modes of transmission and safer sex. There is also no automatic relationship between levels of knowledge and behaviour. Hickson et al. note that whilst UK samples of gay men consistently demonstrate very high levels of knowledge about HIV risks, a significant proportion practice unprotected anal intercourse with regular partners.

Another form of `knowledge and attitudes' research which may be more relevant is the evaluation of skills acquisition, although this can be difficult to measure realistically. The skills acquired might be condom use or the ability to raise the topic of safer sex with prospective partners, and skill acquisition must be measured by self–report.

The final form of attitudinal research is investigation of responses to published materials, media campaigns or particular issues identified as important to the design of prevention initiatives. Examples include evaluation of leaflets by focus groups or assessment of comprehension of core messages by questionnaire or focus group. Such investigations may be necessary in order to establish the acceptability of a certain type of intervention for a target group, or to test an assumption about methods of communication or influence. However, information derived from qualitative evaluations of this sort tends to be much more ambiguous than quantitative data, and contrary to the traditional view that `you can use statistics to prove anything', it is arguable that you can use qualitative research to prove anything you want it to prove, depending on how you interpret what is said to the researcher.

Numbers of people reached

Numbers reached by an intervention are a very basic measure of its success, but a high contact rate does not lead automatically to large measurable effects on behaviour or incidence. A project which reaches a relatively small number of people may have a much greater long–term effect on behaviour and incidence. Measures of quantity are not very useful unless they are accompanied by measures of quality.

It is also useful to focus on demographic groups such as women or young people if research evidence suggests that these groups are at particular risk of HIV infection. For example, there is some evidence that younger injecting drug users are at greatest risk of HIV infection, and it may be appropriate to set targets for contacts with this group as a surrogate or mediator for an effective intervention to reduce HIV transmission amongst injecting drug users. However, there is no clear evidence that young gay men are at greater risk of HIV infection than those over 30, or that black or Asian men are at intrinsically greater risk purely as a consequence of demographic factors, so a demographic target for this group would be inappropriate as one of the mediators for reduced HIV transmission.

Changes in uptake of condoms or injecting equipment

One way of measuring whether an intervention is working is to look at changes in the uptake of condoms or injecting equipment over time. This form of measurement is relatively easy to carry out, since it only involves monitoring the output of a project. However, there is no guarantee that the output of a project can be related to changes in behaviour or incidence.

For example, whilst a project may experience a 25% increase in demand for condoms over the course of a year, there may be no change in the incidence of gonorrhoea in the locality. This might be due to the fact that individuals are ceasing to obtain condoms from any other source, and relying on a project to provide for their total needs. Or the increase might be explained by an increasing uptake of condoms by those who travel in from other districts and who use sexual health services in other districts. Interventions targeted at gay men carried out by individual London health authorities which are measured locally are a good example of this problem (Kelley).

A variant of uptake measurements is the assessment of returns of injecting equipment to syringe exchange projects.

Changes in addiction and treatment patterns

There is substantial evidence that addiction to heroin and drug treatment are strong predictors of HIV risk and risk reduction respectively (Rhodes). Measurements of changes in the time that elapses between the onset of addiction and the first demand for treatment, and of changes in drop out rates from treatment programmes are likely to be important indicators of prevention efforts with injecting drug users. However, it is important to be aware that certain types of treatment programme are in themselves likely to be more successful. Programmes which permit long–term maintenance rather than graduated reduction in dosage were most likely to be successful.

Changes in selfreported behaviour amongst a cohort or sample

Much of the evidence regarding the effectiveness of HIV prevention measures comes from studies of prospective cohorts. These groups of people are recruited at the beginning of an intervention and followed through the study period to assess changes in behaviour. Such studies are reliant on the self–reported sexual or drug–using behaviour of participants, and also carry the risk that participants will be lost to follow–up, thus biasing the results towards the more co–operative or compliant participants.

However, a range of techniques has been developed by behavioural researchers to reduce these potential biases.

For example, Project SIGMA, a UK investigation of gay men's sexual behaviour, used two methods to elicit information about reported sexual behaviour. One was the standard questionnaire method; the other was the process of keeping a sexual diary over the period of a month. Significant discrepancies in self–reporting were noted when the two accounts were compared by researchers (Coxon).

An example of a study in which changes in self–reported behaviour were the measured outcome is the investigation of community diffusion amongst gay men in three small US towns by Kelly et al. This study, described in more detail in Community mobilisation, used measures such as frequency of condom use and instances of anal intercourse.

Changes in reports of sexually transmitted infections

Gonorrhoea incidence has been used as a surrogate marker for unprotected sex by a wide range of researchers. Gonorrhoea incidence is a very responsive marker of changes in sexual behaviour because of the short incubation period of the infection. Active gonorrhoea is also implicated as one of the factors which increases the risk of HIV transmission, so it is reasonable to assume that a fall in gonorrhoea incidence will influence HIV incidence too. A 1994 study in Tanzania demonstrated a strong association between reduced gonorrhoea incidence and reduced HIV incidence by dividing a district into two and pursuing an aggressive screening and treatment programme for STDs in one area, whilst continuing standard HIV prevention activities in the other (Grosskurth).

Between 1993 and 1995 the US state of Louisiana distributed 21 million free condoms in communities defined as high risk (those with highest HIV prevalence and gonorrhoea incidence). Gonorrhoea reports declined 22% statewide during the programme, and researchers noted a strong association between the highest density of free condom outlets, numbers of condoms distributed and greatest decline in gonorrhoea reports when they assessed trends on a district by district basis (Cohen).

This programme was accompanied by a cohort study (n = 620) which compared changes in self–reported condom use between intervention and non–intervention districts. Condom use rose by 14% in intervention districts and 7% in non–intervention districts. The study was not designed to test either the validity of self–reports (for example by cross–checking with partners), or to distinguish between levels of condom use amongst those with many sexual partners and those with few sexual partners.

Nevertheless, at a population level this study offers proof of the concept that social marketing/free distribution of condoms has a significant impact on sexual health which may contribute to HIV prevention.

Changes in HIV incidence or prevalence in the population

HIV prevention workers and researchers have often pointed to changes in HIV incidence or prevalence as evidence that HIV prevention efforts are succeeding or failing. However, one should be very cautious when drawing on such data.

For example, it is extremely difficult to tease apart the range of factors which may be responsible for changes in incidence over time. The most controversial example is the question of whether changes in HIV incidence amongst gay men during the 1980s were a consequence of changes in behaviour or an inevitable feature of the normal pattern of an epidemic.

Whilst it is clear from international data that early safer sex education coincided with rapid falls in sexually transmitted infections, and that HIV incidence peaked in gay communities in 1983/84, it has been argued that the subsequent decline in HIV incidence may be attributable to a declining number of men in the primary phase of infection capable of transmitting HIV easily to their partners. According to this model, a slight reduction in the transmission rate at a relatively early stage in the epidemic would have a disproportionate effect on the multiplication rate of the epidemic. For example, if the average number of partners per month amongst a group with 10% HIV incidence per annum fell by 50% (from 8 per month to 4 per month), the chance of encountering an individual recently infected with HIV would fall by a correspondingly greater multiple. Thus relatively minor and short–term interventions – such as closing bathhouses – may have had a much greater long–term effect than the widespread adoption of safer sex, providing that they occurred relatively early in the course of the epidemic.

However, this model depends on the assumption that infectivity is very high during primary infection (assumed to last for 3–4 months after exposure), and much lower thereafter (Koopman). Recent research in Thailand suggests that this is not the case, and viral load testing has demonstrated wide intra–personal variations in seminal, plasma and vaginal viral load.

Changes in prevalence amongst some segments of the population may indicate that HIV prevention efforts are succeeding in a broad sense, or may give indications of increasing transmission rates. However, it takes a leap of faith to correlate such changes with prevention programmes unless a long timespan is being used. For example, it is reasonable to argue that a fall in HIV prevalence amongst Ugandan women attending antenatal clinics for the birth of their first child suggests that prevention efforts have reduced HIV prevalence in that country, since this group of women are likely to have become sexually active since the beginning of the AIDS epidemic (Asiimwe–Okiror). But a fall in HIV prevalence in the course of one or two years in a much smaller locality cannot be attributed to the efforts of prevention activities in that locality. For example, a fall in HIV prevalence was noted amongst gay men attending GUM clinics in London by the unlinked anonymised HIV prevalance survey (UA) between 1990 and 1995. However, researchers noted that it would be wrong to assume that this reflected a fall in incidence amongst gay men in the capital. The fall in prevalence could be explained by a change in policy at one of the GUM clinics participating in the survey; diagnosed HIV–positive men were no longer treated in the same GUM clinic sessions as undiagnosed gay men, and would not be routinely tested for HIV as part of the UA survey.

Changes in HIV diagnoses are a similarly unreliable guide to the success of prevention efforts, although presentation for a voluntary HIV test appears to be remarkably responsive to media coverage. In 1997 some reports suggested that new HIV cases were rising amongst gay men on the basis of an increased number of HIV diagnoses in the preceding year. Critics of these reports pointed out that the degree of change was very small, and was more likely to represent testing in order to access improved antiretroviral treatment.

Changes in incidence amongst a cohort

A measure which can give a crude measure of the generalised effect of prevention efforts over time is HIV incidence in a prospective cohort. Two examples of such cohorts are the SIGMA gay men's cohort in the UK, and cohorts of injecting drug users recruited by researchers monitoring the success of city–wide needle exchange projects in North America. Data derived from these cohorts demonstrates the strengths and weaknesses of this measure.

The SIGMA cohort reported an increase in HIV incidence amongst its sample in 1990–91 following relatively stable incidence since the cohort began testing in 1987. However, it is important to note that the SIGMA cohort was a `decaying' cohort, with a high drop out rate. Such incidence trends become indicative rather than authoritative in cohorts over long periods of time.

Syringe exchange incidence studies have generally demonstrated a decline in incidence. However, a study in Montreal, Canada, demonstrated a greater risk of seroconversion amongst syringe exchange users than non–users during a mean follow–up period of 15 months (33% vs 13%) (Bruneau). However, another study of all needle exchanges in the Canadian city of Vancouver showed that needle exchanges tended to attract those injecting drug users already identified by other studies as those at highest risk – unstable, high frequency injectors with multiple risks including sex work, unprotected sex with other IVDUs, polydrug use and high frequency of sharing with strangers. Injecting drug users who used syringe exchanges less frequently (less than once a week) were less likely to share these characteristics (Archibald). These two studies of syringe exchange illustrate the danger of jumping to unwarranted conclusions solely on the basis of incidence data.