Holford Watch: Patrick Holford, nutritionism and bad science

Professor Patrick Holford of Teesside University and Head of Science and Education at Biocare and Drew Fobbester are joint researchers and authors of the Food for the Brain Child Survey, September 2007 (pdf). The report lists a distinguished Board of Scientific Advisors for Food for the Brain (FFTB) and thanks them for their overview of the data and analyses. Dr Rona Tutt’s foreword is ambitious:

It is in everyone’s interest to know more about how diet can exacerbate or ameliorate certain conditions and a child’s capacity to learn and to behave appropriately. The Food for the Brain Child Survey is the first of its kind to investigate specific food groups and their association with health, behaviour and academic performance. While the results do not claim to be definitive, the survey is an important step in helping to clarify what optimum nutrition really means for children.

The Food for the Brain Survey should be studied by all those who are interested in children’s mental and physical health and their general well-being. It has particular relevance to those who are in a position to influence national policy. [pg 5]

Inevitably, the caution of this foreword was diluted, even in FFTB’s own summary of the report:

The results, which found a significant association between diet, behaviour and SAT scores, show the shocking state of children’s diet and mental health in Britain with more than 1 in 3 suffering from attention or concentration problems and mood swings or tantrums, with almost half having constant sugar cravings…
Holford concludes: “Failures in government initiatives to improve academic performance may be because we’re not putting money where our mouth is. This survey provides strong evidence that an optimal diet, with more vegetables, oily fish, nuts, seeds, fruit and wholefoods makes a big difference.”

Holford Watch is not entirely sure that it agrees with Holford in this matter. Further, we are not sanguine about the quality or generalisability of the FFTB Child Survey¹ and will explore the issue in some detail in a series of related posts: a series that we may end up calling, The Tedious Elaboration of Observations from a Dubious Survey of Limited Utility.

Even within the report there are fragments that acknowledge that the children in this survey are not representative of children in the UK¹ yet the report was widely and uncritically reported as a snapshot of the diet and mental health of UK children. The present FFTB Child Report summary is an improvement on the previous version, but, like the report, is badly written. Some journalists and commentators mistakenly reported that the findings of the survey had been used to design a diet that had then been tested in the school projects (e.g., Chineham Park Primary, Part 2; Part 3); they discussed the survey’s findings as if they were both the outcome and foundation of a prospective trial. One particularly egregious example of this confusion and overclaiming for the report’s findings is the GMTV Lorraine Kelly piece.

Dr Hilary Jones² discussed the FFTB report on LK Today (WMF; QuickTime). Jones claims that the results of the survey were used to identify a diet which was then used in the school projects. There is an exchange between Lorraine Kelly and Jamil’s mother in which they say that the changes in Jamil were a result of dietary improvement. Dr Jones adds (video 4:10):

It was the results of the survey-the results of the information from 10,000 parents gave the Foundation-that Jamil was put on in his school, this new regime.

This misunderstanding may shed light on his earlier enthusiasm for the results and his unwarranted conclusions about the robustness of the criteria (video 2:33):

I think people have long suspected that good food means good behaviour and bad food’s the opposite. But we haven’t know how much….The association between food and [children's] behaviour at school and their academic achievements at school was actually very strong. There was a very strong correlation between the foods that we would want our children to eat less of-processed food, sugary foods-and the good foods, particularly like fresh vegetables, seeds and nuts-children who ate more of those, actually behaved much better than the other children. They actually got better SAT results at school as well…These were really good, robust criteria for measuring the effect of diet on children’s behaviour. [Emphasis added.]

Holford Watch has outlined some potential confounders that make Jones’ claims for the criteria look rather premature; e.g., socio-economic status and family status. There is the wider issue that the sample of children in the report is not necessarily representative of the UK population of children¹. We will return to possible confounders in another section of this multi-part review.

Lorraine Kelly asks a token question about conflict of interest². Jones gives an interesting reason for discarding any such concerns about the report and its funding (video 5:15):

[Food for the Brain] have got a very robust Scientific Advisory Panel who wouldn’t have been happy any other way. So, I think this survey is scientifically sound, it’s very interesting. And now further work needs to be done to focus down which particular foods and why are the bad foods bad. What actually happens in the brain when you eat processed food or fried food that stops people being able to concentrate and pay attention?

As an intellectual exercise, Holford Watch wonders what would persuade Jones to change his mind about the survey. At present, it looks like he may have been persuaded to believe in the robust science of a report because of the names associated with it rather than the merit of the report itself. If the report and the authors’ conclusions were demonstrably flawed, whether or not it was approved by the Scientific Advisory Board, would this be a strong argument against throwing more money after bad and funding further research to be carried out by those researchers and scientists?

It is widely acknowledged that there are serious problems with peer review³. Reviewers rarely have the time and resources to make a full appraisal of the accuracy of a literature review, the integrity of data and the appropriateness of any statistical analyses etc. even when the subject is their field of expertise. When a paper or report is a little outside an expert’s own field, then there is a tendency to take the paper on trust and restrict your own comments to the conclusions, style and any gross methodological errors. Lamentably, from time to time, this means that reports and papers have never been submitted to full critical appraisal before they are published.

It is important that reports for which large claims are made should be subject to a thorough critical appraisal, particularly when it is hoped that such reports might influence public policy makers. Holford Watch is willing to make a small contribution to that appraisal, so this is the first in a series of posts that will look at this report.

Notes
1 Section 3.2 (pg. 7) acknowledges that the children in this survey are not representative of the child population of the UK although adequate attention is not given to these concerns in the report’s discussion section nor in the press release.

This very large self-selected sample delivers powerful statistical correlations between diet, lifestyle, behaviour and academic performance. The sample, while providing a broad cross-section of the public, were not randomly selected nor has the data been weighted to attempt to make the sample representative of any population. The average SAT scores of the sample were considerably below the national average. It is therefore likely that the respondents were more likely to have children with poor behaviour and academic performance. [Emphasis added.]

Section 4.1 (pg. 9)

The average child’s SAT score is 2.97 [in this survey]. Since the average SAT score in the sample is well below the Government’s target of 4+ as well as the national average, it is possible that the differences observed in their behaviour and performance may be more marked than that of the average child.

Given the above acknowledgements, Holford Watch would argue that the findings of this report should not be presented as indicative of the dietary status or mental health of children in the UK; particularly when the sample is not even vaguely representative of academic achievement as measured by SAT scores.

2 Like Professor Holford, Dr Jones endorses the YorkTest range of products, including the IgG test for food intolerance. In that last link, Holford Watch is uncertain whether to be amused or outraged that this self-styled expert panel quotes the House of Lords report that finds that there is inadequate provision for allergy testing within the UK. That is the same House of Lords report (pdf from which pg nos are given) that investigated the issue of direct-to-consumer tests such as the YorkTest foodSCAN IgG test for food intolerance and the YorkTest-Allergy UK MAST IgE test for allergies to food and airborne allergens, amongst others (pp 86-88). The House of Lords report made the following recommendation:

We are concerned both that the results of allergy self testing kits available to the public are being interpreted without the advice of appropriately trained healthcare personnel, and that the IgG food antibody test is being used to diagnose food intolerance in the absence of stringent scientific evidence…We urge general practitioners, pharmacists and charities not to endorse the use of these products until conclusive proof of their efficacy has been established. [Emphasis added.]

Presumably the self-styled expert panel and Dr Jones have strong reasons for ignoring this strong recommendation from the House of Lords. Or, is Dr Jones persisting in believing YorkTest’s notion that there is any such evidence, despite expert review of these issues by both the House of Lords and the ASA? If so, it would behove him to explain why these independent reviews should be rejected in favour of his opinion. If there is no such explanation, Holford Watch is a little concerned that this may compromise the value of Jones’ assessment of the “sound science” of the FFTB Child Report.

Holford Watch has some sympathy for the notion that you can not interrupt the smooth flow of television by asking for a declaration of a conflict of evidence on every matter but it might have been interesting if Lorraine Kelly had commented upon the fact that YorkTest has made contributions to FFTB and that both the FFTB Chief Executive Patrick Holford and Dr Jones endorse the YorkTest products.

3 There was the notorious flimflamming of JAMA by TM advocates. There was the sadly realistic 1997 editorial by Richard Smith on the matter in the BMJ:

One piece of evidence we did have from earlier research was that blinding reviewers to the identity of authors improved the quality of reviews,(1) but three larger studies presented at the congress found that it did not. The new studies also found that blinding was successful in only about half to two thirds of cases. One of those studies - by Fiona Godlee from the BMJ and two colleagues - might also be interpreted as showing that peer review “does not work.” The researchers took a paper about to be published in the BMJ, inserted eight deliberate errors, and sent the paper to 420 potential reviewers: 221 (53%) responded. The median number of errors spotted was two, nobody spotted more than five, and 16% didn’t spot any.

This study is indicative of the problem that even when a paper does like within a reviewer’s area of expertise, it doesn’t follow that the same reviewer will be able to comment knowledgeably on other aspects of that paper or capable of detecting errors.

It is a nuisance, but it would seem that the only compromise for now is for reviewers and advisors to adopt the increasingly popular journal practice of describing which author did what. So, for example, a reviewer might say that he/she read the literature review for sense but did not check any of the primary source material. Similarly, that he/she had commented on the conclusions drawn from data, but had not checked the integrity of it, nor verified the robustness of the statatistical analyses etc. Only then can the reader be certain that a report reflects the knowledge and integrity of those who are said to endorse it.

Update to [3]: Matt Hodgkinson has an outstanding discussion of issues involved in the peer review process: A case study in open peer review. Given the legendary spats of some academics however, even open and transparent processes like this may have their pitfalls.

Further Reading

Food for the Brain Child Survey 2007: The Promotion
Holford Watch looks at the literature review:
Food for the Brain Child Survey 2007: Review Part 1
Food for the Brain Child Survey 2007: Review Part 2
Food for the Brain Child Survey 2007: Review Part 3
Food for the Brain Child Survey 2007: Review Part 4
Food for the Brain Child Survey 2007: Review Part 5

Holford Watch appeals for help to Professor Holford and two members of the Scientific Advisory Board who approved this report and then looks at the data and analyses:
Food for the Brain Child Survey 2007: Review Part 7
Food for the Brain Child Survey 2007: Review Part 8
Why Don’t Food for the Brain Report Their Survey Results on Supplement Pills Survey: Review Part 9
Food for the Brain Child Survey 2007: Review Part 10