Health & Medicine

Cochrane’s incomplete and misleading summary of the evidence on deworming

Temina Madon

The Cochrane Collaboration recently updated their review of the evidence on treating school-age children for soil-transmitted helminths (STH, i.e. intestinal worms which infect nearly a quarter of the world’s population). While many of the methodological flaws in the 2007 review have been addressed, the new review is still incomplete and provides a misleading summary of the evidence on mass deworming. Deworm the World, Innovations for Poverty Action (IPA), the Center for Effective Global Action (CEGA) and the Abdul Latif Jameel Poverty Action Lab (J-PAL) all encourage countries with moderate to high worm loads to mass treat for STH because it is one of the most cost-effective ways of increasing school attendance. These results remain unchanged and more recent studies (see below) have only strengthened the case. We have not yet carefully reviewed Cochrane’s summary of the evidence on height, weight, anemia, and cognitive outcomes (although others have commented), but the educational benefits alone are sufficient to justify mass school-based deworming.

Problems with the Cochrane review: While claiming to cover published and unpublished studies the review excludes three important randomized and quasi-randomized studies on STH which all show positive results. Bleakley (2004) shows that an early 20th century campaign to eradicate hookworm in the US south improved school attendance, literacy, and income in adulthood. Ozier (unpublished) shows that children under one year of age (and therefore too young to be treated) at the time of mass deworming in their communities in Kenya had significantly improved cognitive outcomes due to spillover effects.  Baird et al (unpublished) shows strong long term labor market outcomes of children treated for STH.

Miguel and Kremer (2004) is the only study quoted in the main results section of the Cochrane review with outcomes on student attendance (the other quoted studies are not clustered and therefore appropriately ignored). Yet the review authors down-weight the Miguel and Kremer results on student attendance for a number of reasons. For example, that there is “high attrition” in hemoglobin (Hb) data, even though there is zero attrition for the actual student attendance results!  (The reason for the smaller Hb sample is that it was only collected for a random subsample, information that is readily available from the authors).

The Cochrane authors also criticize Miguel and Kremer (2004) for a lack of baseline (pre-treatment) data on school attendance. This is a strange point for at least four reasons. First, given the experimental design there is no systematic reason to expect differences in school attendance between the treatment and control groups. Second, Miguel and Kremer (2004) show that there is baseline balance along a wide range of other characteristics in the treatment and control groups (in Table 1), many of which correlate strongly with school attendance (i.e., family socioeconomic status). But most importantly, there is in fact baseline “balance” between Group 2 and Group 3 in the high-quality school attendance data in 1998, when both were still “control” before Group 2 was phased into deworming (see Table 8 of the Miguel and Kremer 2004 paper for this data). The Cochrane authors apparently missed this in their read of the paper. Finally, the Cochrane authors strangely “slice” up the data into impact in the first year after deworming versus the second year, and then across different “comparisons” of the treatment groups. Measuring impacts in small subsamples of the data inevitably reduces the precision of the estimate, making an effect that is strongly significant in the full sample look insignificant in small slices.

Missing the point about mass deworming. In their conclusion, the review’s authors state that “Screening school children for intestinal helminths and then treating those infected probably has some value…[but] there is insufficient evidence to recommend deworming drugs in targeted community programmes.” This argument overlooks the fact that if a program is effective for individuals with worms, it will still affect the same individuals (presumably with the same effect) if they are reached through a mass treatment campaign. The only reasons to prefer a screening approach is if deworming drugs had negative effects on uninfected children (they do not), or if the costs of treating uninfected children in a mass campaign were greater than the costs of individually testing children to determine whether they required treatment (in fact it is much cheaper to mass treat than to diagnose and treat).

School attendance is an important goal in its own right but it is also a step towards longer term outcomes. Preliminary results from Baird et al suggest that mass treatment for STH leads to longer hours worked and higher earnings 10 years after treatment — results remarkably similar to those found in the US South. We therefore strongly endorse the WHO and Copenhagen Consensus’s recommendation to mass treat children for STH.

This post is co-authored by Annie Duflo (Executive Director of IPA), Alissa Fishbane (Managing Director of Deworm the World Initiative, IPA), Rachel Glennerster (Executive Director of J-PAL Global), Michael Kremer (Gates Professor of Developing Societies in the Department of Economics at Harvard University), Temina Madon (Executive Director of CEGA), and Edward Miguel (Professor of Economics at UC Berkeley).

Bookmark and Share
Comment to "Cochrane’s incomplete and misleading summary of the evidence on deworming":
    • DTR

      Cochrane’s independent appraisal of reliable evidence

      We thank the authors for their response to the updated Cochrane review:
      http://onlinelibrary.wiley.com/doi/10.1002/14651858.CD000371.pub4/abstract;jsessionid=586D3310BB0C680B7CEBACBF32DEA121.d03t01

      We note their concerns about our analysis and we respond below. However, statistical intricacies related to one outcome from one trial need first to be set in context. As Cochrane authors our motivation lies solely in providing an independent appraisal of the available evidence about the effects of health treatments, policies and public health strategies. The current deworming strategy, as promoted by the WHO and multiple advocacy groups, is widely claimed to be the ‘best spend’ to improve school health, school performance, increase productivity and reduce poverty. Our review systematically identifies, appraises and summarizes the available evidence from randomized controlled trials to support or refute these claims. We seek evidence of these desired impacts on child mortality and school performance; and evidence on important outcomes required to achieve these desired impacts, including nutritional indicators, haemoglobin, cognition and school attendance.

      Our review shows that the evidence on desired impact (child mortality and school performance) is absent. The evidence on the important outcomes is not strong, being largely confined to evidence of improved weight gain in three trials conducted over 15 years ago. Two of these trials were carried out in a specific setting in Kenya by the same team (Stephenson et al) in an area with very high worm loads – probably, we would surmise, far higher than in most settings where programmes are promoted today. The third trial that showed important weight gain was from urban India. Subsequently conducted trials – some of them very big – have largely either failed to demonstrate benefits, or have not been published. The latter includes a trial of one million children completed in 2006. Importantly, this trial, for which no data are currently available, was carried out at the same site in India as the earlier study showing an effect on weight gain.

      To recap the findings of the Cochrane review: apart from a mixed effect on weight, with three older studies showing effects, our analysis of the global evidence base shows there is fairly good evidence of no impact on haemoglobin; no reliable evidence on cognition or school performance; and the evidence for school attendance is limited.

      We turn now to the criticisms in the blog:

      • The comment that some trials are missing. Our analysis was limited to randomized controlled trials comparing mass administration with placebo or no treatment. The three additional studies cited in the blog do not meet these criteria and so were appropriately excluded.

      • The quality of the evidence for school attendance was low. This was reached using standard GRADE criteria, including a high risk of selection bias (due to inadequate methods of generating the random sequence and concealing treatment allocation, and the lack of baseline data), and the statistical uncertainty given that the result did not reach standard levels of statistical significance (95%).

      • In relation to baseline values for school attendance in the Kenya study, the reporting of relevant baseline data is standard practice for randomized trials, and a requirement by the CONSORT Group (http://www.consort-statement.org/consort-statement/13-19—results/item15_baseline-data/). Small differences in baseline values can cause spurious conclusions with small numbers of units of randomization if only end values are presented rather than change values. It remains possible that the differences in end values reported are the result of modest baseline differences.

      • The blog comments that we split data across years and hence reduce power. We did indeed present data before one year and after one year to allow us to use as much data as possible in these time periods, since some trials present data at multiple follow up periods, and to help detect any externalities of the intervention in trials with longer follow-up. A combined analysis across the two time periods does not substantively change the results or conclusions.

      • Children known to be infected with worms are usually treated – this is what happens in clinical practice or in screening programmes. We personally would prefer that children were not infected with worms as they are unpleasant. But the policy under debate here is about the effects of mass deworming programmes and whether these demonstrate an effect at the population level. Our review collates the best available evidence on this, as we have summarized above.

      Evidence-informed health policy, that we have been part of over the last 15 years, means we need to consider the global evidence base, not make policies based on individual trials or studies. Our review is complete in this regard, apart from any studies that remain unpublished and outside the public domain. The review raises questions about the generalizability of the results on school attendance from this single trial by Miguel and Kremer, given the lack of nutritional and cognitive effects in our systematic review of all reliable studies. Inevitably this also leads to questions about the plausibility that worms are a major and widespread cause of absenteeism at school, and whether mass deworming programmes have the kind of impact on school performance and development that is claimed by the advocates.

      David Taylor Robinson, Sarah Donegan, David Sinclair, Paul Garner (30th July 2012)

      [Report abuse]

Leave a comment

 

 

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>


+ 8 = 13