Holford Watch: Patrick Holford, nutritionism and bad science

You may remember that Miriam Barry of the Irish Association of Nutritional Therapy (IANT) offers a Response to the recent media coverage regarding antioxidants. She opens her response with these words:

As nutritional therapists we feel compelled to give the public the facts of this case. Please click here to inform yourself of the facts regarding this study.

We left a comment on Tuesday but it still hasn’t appeared (between flaky software and spam filters, things do go awry). Having read Barry’s characterisation of the Cochrane Review, it does not match our impression of what the review says and we have been through that report several times. Nonetheless, we may have missed something so offer some comments here (we can’t cover all of Barry’s errors or misdirections) and refer you to Dr Ben Goldacre’s recent article where he tackles some of the new urban myths about this review:“Manufacturing Doubt”: Sir Cliff Richard weighs in on the Cochrane review.

Barry makes these remarkable criticisms:

[The Cochrane] review is at its best completely flawed and at its worst seriously damaging and misleading. The authors have excluded over 400 trials many of which have a positive outcome and have no deaths so if these were included the findings would be different…

Many of the studies involve diseased or high risk groups so the findings are not relevant to healthy populations which the authors state themselves.

To address the last point first. Looking at the overall study:

Twenty-one trials included 164,439 healthy participants. Forty-six trials included 68111 participants with various diseases (gastrointestinal, cardiovascular, neurological, ocular, dermatological, rheumatoid, renal, endocrinological, or unspecified). [quoted from the abstract.]

So, it looks as if healthy participants make up around 71% of the total pool of participants (232,550).

For those with access to the full review (pdf), there is more detail.

Twenty-one trials were primary prevention trials including 164,439 healthy participants; 46 trials were secondary prevention trials including 68111 participants with gastrointestinal (n = 10 trials), cardiovascular (n = 9), neurological (n = 6), ocular (n = 5), dermatological (n = 5), rheumatoid (n = 2), renal and cardiovascular (n = 1), endocrinological (n = 1), or unspecified (n = 7) diseases. [pg. 8 pdf, pg 5 document.]

The primary prevention trials involved adults who were healthy and were recruited from the general population. The secondary prevention trials involved adults who had chronic diseases in a stable phase. Some of the chronic diseases were cataracts and such as well as the more obvious chronic diseases that have more familiar multi-system effects.

The reviewers excluded trials that involved pregnant or lactating women, children, or those with acute infectious illnesses or active malignant diseases. We would imagine that there is no difference of opinion as to the benefits of vitamins and minerals for acknowledged deficiencies of essential nutrients, or that there may be particular life stages where these are more common (such as pregnancy).

We can’t find the place in the report where Barry asserts that the authors say that their findings are not relevant to healthy people. We can find places where Bjelakovic et al remark:

Our review represents a comprehensive review of the topic, including 67 randomised trials with almost a quarter of a million participants. This increases the precision and power of our analyses… [pg 12 pdf; pg 9 document]

Oddly enough, the authors think that one of the limitations of their study is that:

These populations mostly came from countries without overt deficiencies of specific supplements. Accordingly, we are unable to assess how antioxidant supplements affect mortality in populations with specific needs… [pg 13 pdf; pg 11 document]
Furthermore, we did not examine the treatment effect of antioxidant supplements (tertiary prevention) in specific patient groups or the preventive effects of antioxidant supplements for patient groups with verified specific need for antioxidant supplements. Other systematic reviews should address these issues…
We found no convincing evidence that antioxidant supplements decrease mortality…Therefore, we cannot recommend the use of antioxidant supplements as a primary and secondary preventive measure in the population groups studied in the present review. [pg 14 pdf; pg 10 document; emphasis added]

Barry claims that:

• “Many of the studies are with dosages that far exceed what is in most vitamin products” which would be unsettling if it were an accurate interpretation of the report. You can consult the reviewers’ table of RDAs, experimental doses etc. in figure 4 (pg, 162 (pdf), pg. 159 document). See Goldacre who provides detail of the dosages available in his local Holland and Barrett: not to spoil the reveal, but it seems that these ‘high doses’ are routine. And, it seems that no lesser an august institution than the Orthomolecular Medicine Newsletter has this to say about ‘vitamin myths‘:

  Since vitamin myths persist, the facts of orthomolecular medicine must be presented in straightforward, memorable terms, such as:
  “The number one side effect of vitamins is failure to take enough of them.”
  “Negative vitamin studies use low doses. Positive vitamin studies use high doses.” [Emphasis added.]

  That would be the orthomolecular or optimum nutrition approach that underpins most of the non-evidence based teaching of nutrition qualifications and courses. jdc325 provides some useful information about David Adams of the HFMA (source of many of the stories that criticise this Cochrane Review) and his campaign for high dosage supplements.

• “[The reviewers] state that 5 portions of fruit and vegetables a day should supply you with sufficient nutrients”. However, we can’t find that statement in the review and would be grateful for the page reference.
• “The final exclusion involves the trials with selenium with no reason as to why. Selenium trials generally show a positive result on the disease in question when supplemented.” Eh, selenium wasn’t excluded from all the analyses as you will see on pg 12 of the pdf (pg 9 of the report). The authors provide their explanations in several places. And, in fact, it is the Cochrane reviewers who state:
  

  Selenium used singly or in combination with other antioxidants significantly decreased mortality when including all 20 trials…Selenium had no significant effect on mortality in 14 low-bias risk trials…or 6 high-bias risk trials…
  Selenium tended to reduce mortality but only when high-bias risk trials were considered… [pg 12 pdf; pg 9 document]
  Selenium given singly or in combination with other supplements seemed to significantly decrease mortality, but after exclusion of high-bias risk trials, the effect disappeared…Results of ongoing randomised trials with selenium will likely increase our understanding of the effects of selenium…Recently, a randomised trial and an observational study have shown that selenium may carry health risks…Therefore, current and any future trials should be monitored closely for harm. [pg 14 pdf; pg 11 document]

  If a reader still needs to be convinced that the reviewers looked at the selenium trials, then take a look at the funnel plot of the low risk of bias selenium trials: figure 3 (pg, 161 (pdf), pg. 158 document). And there seems to be an analysis of trials with selenium in analysis 01.12 (pg, 186 (pdf), pg. 183 document).

When there is so much that Barry could comment on, it is rather wasteful to highlight issues that aren’t there. Finally Barry asserts that, “There are many quotes from experts in the field of science who have reviewed this paper and seriously question the findings due to its exclusions”. It’s our impression that it would be possible to find many quotations from scientists who have read the Cochrane review and would support it as a well-conducted, gold-standard review. If this is not to be an example of “Our experts are bigger than yours” then it would be helpful if the ‘informed criticism’ gave some well-referenced examples of specific criticisms.

When scientists consider inconsistencies in the meta-analysis supporting a set of findings, they typically do not reject the entire systematic review out of hand, as manufactroversy provocateurs do. Anomalous data or experimental results are an opportunity to improve or modify the understanding of a phenomenon or the ideas that underpin it. The manufactroversy provocateurs would do better to pool their resources and come up with a systematic review that is as transparent and well-conducted as the Cochrane. If they were to do that and report markedly different results, then that would be profoundly interesting.

The takeaway message from this review is that, contrary to expectations and the general thrust of the anti-pathological benefits of the antioxidant hypothesis, antioxidants supplements did not result in a reduced mortality rate.

As we said, there were too many errors to deal with all of them, and there is little value in pursuing Barry’s red herrings: we have done our stint as Summer Glau.

Update 27 April: Oddly enough, IANT did not indulge in this Holford canard but we see it so often that we have decided to include it here for completeness: Are supplements harmful?

In this review, which is a rehash of their paper published last year in the Journal of American Medical Association (JAMA), they first excluded over 400 trials, many of which had positive outcomes, but had no deaths. They then decided which trials they liked (low risk bias) and did not like (high risk bias), a factor that has received criticism in mainstream medical journals

We responded earlier and a very patient Professor Gluud corrected Holford in the course of a Radio 5 item.

Essentially, Bjelakovic et al discuss their exclusion of 405 trials. These trials involved 40,000 participants: they were mostly small Phase I or II trials and short-term with little or no follow-up and no clinical outcome measures (pg. 8 (pdf), pg. 5 document). Bjelakovic et al contacted all of the authors to enquire whether there were any deaths. Around 20% of the authors replied and reported that there were no deaths in either arm of their studies. The authors did not exclude studies that were positive for the supplement arm but negative for the placebo or control arm.

Several people have expressed concern that eliminating the 405 studies “create[s] a bias in which the therapy on trial appears to be associated with deaths?”. However, these trials were, as the authors described above and only involved 40,000 rather than the 232,550 participants in the 67 trials that were assessed: 164,439 of these participants were healthy. You can read about the characteristics of the excluded studies in the review (pp. 118-42 (pdf), pg. 115-39 document).

The authors did take account of the 405 trials by performing a sensitivity analysis.

Therefore we performed exploratory analyses adding an imagined trial with 1 death and 20000 participants in each intervention group. The influence of zero events trials on our final result was not noticeable.

Their findings didn’t change.

References

Bjelakovic G, Nikolova D, Gluud LL, Simonetti RG, Gluud C. Antioxidant supplements for prevention of mortality in healthy participants and patients with various diseases. Cochrane Database of Systematic Reviews 2008, Issue 2. Art. No.: CD007176. DOI:
10.1002/14651858.CD007176.
We have dealt with some of the content of the more common puzzling ‘criticisms’ in these posts:
Patrick Holford and His Own Reality: Part 2, estimating risk bias in Cochrane reviews
Catherine Collins: “Patrick has [given] an absolutely perfect example of why one should be wary of nutritional therapists.”
Patrick Holford and Contriving a Controversy: the Cochrane review of antioxidant supplements