Holford Watch: Patrick Holford, nutritionism and bad science

Back in January we wrote to Professor Patrick Holford of Teesside University, Head of Science and Education at Biocare and CEO of Food for the Brain: we asked some questions about the survey to help us perform a robust review. We waited for three weeks but did not receive any responses and, thus hampered, continued to review the survey and uncovered about as grisly a work of ineptitude with statistics as has ever come our way.

The FFTB Child Survey literature review was irrelevant and incompetent. But the number-crunching and display of summary data were breathtakingly, unbelievably bad. Office-neighbours-should-have-been-pounding-on-the-wall-and-calling-the-statistics-authorities-and-reporting-a-hazard-to-health bad. The-guilty-parties-should-be-having-their-keyboard-privileges-revoked bad. The authors didn’t even have the courtesy to respect readers’ time and energy by checking basic things like correctly labelled axes for the graphs or even correct titles for the graphs. However, that is nugatory in the overall scheme of the awfulness of the statistical analysis. We’re conflicted. We don’t want you to expose yourself to the full horror of that report yet it may be one of those things that you can’t quite believe until you do look for yourself: we offer a brief summary.

But, what does Holford say about this?

A [Holford's response]. “The Food for the Brain Child survey is a simple survey of the diets, academic performance and behaviour of a group of over 10,000 children, a third of which also provided SAT scores. The survey simply reports which food group consumptions are associated with parental ratings of better or worse behaviour and academic performance, and SAT scores. The conclusions are based solely on this data, and make no reference to any other studies. The survey is available for all to see at our website www.foodforthebrain.org/childsurvey. There is a Q&A section for people to post critiques, and for our Scientific Advisory Board to respond. Professor Colquhoun’s critiques have been responded to. Professor Stein has not informed us of any. If there are errors we do our best to correct them.”

The numbers are irredeemable. There is nothing to be done to salvage them. There are serious issues concerning the survey questions and this is yet another object lesson in why research must be conducted using standardised and validated questionnaires. The data were farcical even before the revelation that Professors Holford, Philip Cowen and David Smith (amongst others) want us to believe that something of statistical significance can emerge from a comparison of 2 outlier groups from an unrepresentative survey of 10,222 children. Outlier groups of 32 and 42 children which, when added together, make up 0.72% of the children. Don’t even trouble your head as to whether there has been a cluster analysis to take account of any children from the same families. Every survey taker wants to highlight 0.72% of their total unrepresentative sample and pronounce on public and research policy on the back of that. However, more about this at another time and if you are interested, you might consult our multi-part review of the Food for the Brain Child Survey 2007 (references below).

We are astounded that Holford claims to be unaware of problems concerning the literature overview in the FFTB report (we don’t know what Professor Stein thinks but we would be surprised if we differ much on this point). The literature overview is spectacularly disappointing: we will repeat our summary here. What is especially depressing about this lamentable overview is that it looks to have been adapted from Chapter 16 of Food Is Better Medicine Than Drugs (FIBMTD) by somebody who had absolutely no degree of understanding for the comparative nuance that the author of that section used, nor for the ballpark relevance of the literature that is cited in the context of a discussion about children with ADHD rather than the general population of children (as per the FFTB Child Survey). We disagree with the extravagance of the claims of “proof” and some of the nuance is wrong but it is peculiarly gratifying to see that, at some point, somebody understood the the distinctive groups of children who participated in the studies. We shall look at the FIBMTD version in 2b because it is instructive and gives a clue as to how some people fail to understand nuance.

This is what Holford and Fobbester wrote in Food for the Brain Child Survey, September 2007 (pdf). Section 3.1 (pg 6):

A substantial and convincing body of scientific evidence, including many randomised-controlled trials, shows that a child’s nutrition has a profound effect on their learning and behaviour:

• Ten out of twelve randomised controlled trials (RCTs) assessing the impact of vitamin and mineral supplementation have illustrated significant improvement in non verbal IQ and concentration^[1]
• A diet high in essential fats, especially omega 3 essential fats, as well as supplementation improves attention and reading, and reduces anxiety and aggression. Numerous studies have been conducted in this area^[2] including two recent randomised controlled trials^[3,4]
• A diet with a balanced glycemic load, incorporating more whole foods and slow-releasing sugars and less refined foods and fast-releasing sugars, eaten at regular intervals, may improve learning, attention and reduce anxiety and aggressive behaviour^{[5, 6, 7]}
• A significant proportion of children with ADHD may have unidentified food and chemical sensitivities^[8,9] Gluten sensitivity, for example, is far more common in children with behavioural problems than in those without^[10]

The evidence from what is now a large and growing body of broadly consistent research links sub-optimum nutrition with poor cognition and behaviour. While much is known about the kind of diet that provides Recommended Daily Allowances, little is known about the kind of diet that equates to optimal mental health, learning and behaviour. Since RDAs largely do not take into account recent research on nutrition and mental health there is no good reason to assume that these kinds of levels, if eaten, are optimal for mental health.

This is what we wrote as a summary of the problems with the literature overview, in January. If you want the gory detail, read the linked posts. But, to summarise:

• Benton’s review^[1] has the potential to be a relevant reference but Holford and Fobbester mis-report the findings as indicative of “significant improvement” rather than “positive response” (it does matter) and they mis-report the number of RCTs. They also fail to notice Benton’s caveats about generalising the results to the wider population of children.
• Holford and Fobbester’s overview suggests that most of the trials to which they refer were conducted in a general population of children, however:
  • Richardson’s review^[2] is about the role and putative benefits of fatty acids for children with developmental and psychiatric disorders and there is no evidence to suggest that her findings can be generalised to the wider population of children who do not have developmental complications
  • the fish oil trials^[3,4] in these references were similarly for a specialised population and not generalisable (as above).
• Holford and Fobbester supply 3 references for the putative value of a “diet with a balanced glycemic load” for children^{[5, 6, 7]} but none of the references address this:
  • Haapalahti et al^[5] discuss the dietary habits of children with functional gastro-intestinal disorders. These results are not representative of a wider population of children
  • Benton’s review^[6] examines the importance of breakfast to later mood and performance. He makes some suggestions about the importance of meal scheduling on the cognitive performance of some of the population
  • Lien et al^[7] discuss the consumption of soft drinks and associations with the incidence of hyperactivity, mental distress and conduct problems in adolescents. They do not discuss ‘glycemic load’, whole foods or the importance of regular meals.
• both Egger et al^[9] and Carter et al^[8] discuss children with hyperkinesis symptoms who have food and chemical sensitivities. However, both of these studies concern children who have co-morbidities; ADHD and hyperkinesis are not necessarily equivalent; and all of the children had been referred to specialist clinics. These findings can not support the assertion about the wider population of children with complex behavioural difficulties, far less have any verifiable and substantial contribution to the study of nutrition among children who do not have these symptoms or disorders
• Holford and Fobbester’s reference for gluten sensitivity and children with behavioural disorders^[10] does not exist.

This looks like a list of assertions for which the authors feel the truthiness, and attempt to create the necessary scienciness through referenciness. It is clear to even the casual reader that Holford was making assertions about the RDAs based upon his interpretation of the literature; a literature that he appears not to cite accurately nor to interpret appropriately.

Holford and Fobbester refer to the truism that a child’s nutrition affects their learning and behaviour. They cite studies to argue that children with “sub-optimum nutrition” (an undefined term) have poor cognition and behaviour. The studies that the authors cite indicate that multi-supplements may be of benefit to a small sub-group of children. However, several reviewers have remarked that these are most effective for children who might have deficiencies that approach clinical levels; even appropriate supplementation is unlikely to promote a substantial difference in the absence of other supportive factors such as a stable homelife or stimulating education.

There is a buzz of excitement about the claimed benefits of Omega 3 supplements. However, it is unfortunate that the evidence in this area is currently of such poor quality that it weakens its probative value. The FFTB authors draw our attention to studies that involve children with clinical disorders; results of such studies may not be generalisable to a wider population.

It is premature, at best, to attempt to use the results of this FFTB report or its literature overview to argue that there is a robust scientific case for a change in public policy or dietary recommendations that affects the general population of children. The results are not even sufficient to sustain the authors’ animadversions about the current recommended RDAs. Ironically, when so few children in this survey of children (albeit, it is not a representative sample) adhere to the dietary recommendations that already exist (see, e.g., the publications of the Caroline Walker Trust), it would seem that this sample can not be used to comment on the adequacy or otherwise of RDAs.

It is disingenuous of Holford to start making modest claims for the influence or meaning of the Food for the Brain Child Survey 2007 when the media promotion and Dame Tutt’s foreword to the report are a marked contrast to such modesty or fine restraint in publicising the findings (findings that are overwhelmingly flawed, in their current forms).

This is the scholar who has a visiting professorship at the University of Teesside. An institution where, one hopes, they teach young scientists to cite research accurately, to interpret it appropriately. Nonetheless, a place where Professor Holford shall be supervising students and applying for research support.

And Holfordwatch has a special appreciation for the disingenuous claim that FFTB now offers a Q&A for posting critiques. We wrote to Professor Holford on January 11 and did not receive any response. Our criticisms of the report have been detailed and thorough, and available since January: our criticisms were known to both Holford and other members of the FFTB Scientific Advisory Board. We are still waiting for answers.

References

[1] Benton D, Micro-nutrient supplementation and the intelligence of children. Neurosci Biobehav Rev. 2001 Jun;25(4):297-309.
[2] Richardson AJ, Long-chain polyunsaturated fatty acids in childhood developmental and psychiatric disorders. Lipids. 2004 Dec;39(12):1215-22.
[3] Richardson AJ, Montgomery P. The Oxford-Durham study: a randomized, controlled trial of dietary supplementation with fatty acids in children with developmental coordination disorder. Pediatrics. 2005 May;115(5):1360-6. [The authors misquote the name in the FFTB Survey.]
[4] Sinn N, Bryan J. Effect of supplementation with polyunsaturated fatty acids and micronutrients on learning and behavior problems associated with child ADHD. J Dev Behav Pediatr. 2007 Apr;28(2):82-91.
[5] Haapalahti M, Mykkänen H, Tikkanen S, Kokkonen J. Food habits in 10-11-year-old children with functional gastrointestinal disorders. Eur J Clin Nutr. 2004 Jul;58(7):1016-21.
[6] Benton D. The impact of the supply of glucose to the brain on mood and memory. Nutr Rev. 2001 Jan;59(1 Pt 2):S20-1
[7] Lien L, Lien N, Heyerdahl S, Thoresen M, Bjertness E. Consumption of soft drinks and hyperactivity, mental distress, and conduct problems among adolescents in Oslo, Norway. Am J Public Health. 2006 Oct;96(10):1815-20.
[8] Carter CM, Urbanowicz M, Hemsley R, Mantilla L, Strobel S, Graham PJ, Taylor E. Effects of a few food diet in attention deficit disorder. Arch Dis Child. 1993 Nov;69(5):564-8.
[9] Egger J, Carter CM, Graham PJ, Gumley D, Soothill JF. Controlled trial of oligoantigenic treatment in the hyperkinetic syndrome. Lancet. 1985 Mar 9;1(8428):540-5.
[10] The FFTB Child Survey cites: Gerarduzzi T et al. Celiac disease in USA among risk groups and general population in USA. Journal of Pediatric Gastroenterology and Nutrition. Vol 31 (suppl) 2000: pp S29, Abst 104. [Having searched Jnl of Ped Gastro and Nutr, this paper doesn't seem to exist as per this reference. It appears in Google Scholar as a citation only which might indicate an error.] Holford Watch has previously mentioned our difficulty with this reference: Running out of tolerance.

Further Reading

Food for the Brain Child Survey 2007: The Promotion
Holford Watch looks at the literature review:
Food for the Brain Child Survey 2007: Review Part 1
Food for the Brain Child Survey 2007: Review Part 2
Food for the Brain Child Survey 2007: Review Part 3
Food for the Brain Child Survey 2007: Review Part 4
Food for the Brain Child Survey 2007: Review Part 5

Holford Watch appeals for help to Professor Holford and two members of the Scientific Advisory Board who approved this report and then looks at the data and analyses:
Food for the Brain Child Survey 2007: Review Part 7
Food for the Brain Child Survey 2007: Review Part 8
Why Don’t Food for the Brain Report Their Survey Results on Supplement Pills Survey: Review Part 9
Food for the Brain Child Survey 2007: Review Part 10