Holford Watch: Patrick Holford, nutritionism and bad science

Professor Patrick Holford of Teesside University (and also Head of Science and Education at Biocare) and Drew Fobbester are joint researchers and authors of the Food for the Brain Child Survey, September 2007 (pdf). Holford Watch was disappointed by the report’s literature overview. We approached Patrick Holford and two members of the Scientific Advisors who had approved the data analysis with our questions about the FFTB report data but we did not receive any answers. Our comments about the data analysis and presentation of the data reflect our confusion.

Holford and Fobbester are so pleased with the graph that displays SAT Score Variance with Overall Diet that they use it twice: Section 4.2.1 (pg. 9) and 10.1 (pg. 33). The careful reader will note that in both places, the authors claim:

There was a direct and consistent increase in SAT scores with improving overall diet, showing that children with ‘very good’ diets have mean SAT scores that are 11% (p [less than] 0.05) than children [sic] with ‘poor’ diets. [pg. 9]

It seems self-evident to note that there is not an 11% difference between the mean SAT scores of the children with Very Good (mean 3.26) and Poor (mean 3.02) diets, this is approx. an 8% difference. There is an approx. 11% difference between the scores of the children with Very Poor (mean 2.93) and Very Good diets. The differences between the diet groups are put into some perspective when the reader considers that Holford and Fobbester report that the government target for SAT scores is 4+; meeting this target would require improvements of between 22-34% for the children in these categories.

Furthermore, if you look at the graph, it is immediately apparent, even from the thumbnail, that there is no “direct and consistent increase in SAT scores with improving overall diet”.^[1] Children with a Poor diet have an mean SAT score of 3.02 which is higher than both the Neutral (mean 2.94) and Good (mean 3.01) groups. The means of children with a Very Poor and Neutral diet may be effectively equivalent at 2.93 and 2.94 respectively. Here we come to an interesting issue because we don’t know if the number of children in these groups are roughly comparable or widely disparate. If the numbers are very different, then they are not robustly comparable. If both groups the Very Poor andVery Good groups have such small numbers that they are effectively outliers, this would call into question why this should be given as a useful comparison.

One might find it helpful if the reader were to understand how many children fall into each category of diet. Holford Watch would have found it very helpful. Sadly, this information is not provided. From the choice of bar graph and the 5 columns of equal width, it might seem as if the numbers in each category might be the same - 5 categories with 20% of the children in each. Holford and Fobbester don’t comment on this on pg. 9, but they add this explanation when they reproduce the graph later.

Based on the apparent positive and negative relationships between foods and academic performance rating, behaviour and actual SAT scores the researchers applied an overall diet score to each individual based on their consumption of each positive or negative food…

Having given an overall diet score to each individual they were grouped into ‘Poor’, Neutral, ‘Good’, and ‘Very Good’. ‘Poor’ diets are high in negative foods and low in positive foods and ‘Very Good’ diets are high in positive and low in negative foods. The worse the overall diet score the worse the SAT scores. [pg. 33]

It is unfortunate that Holford and Fobbester do not explain whether or not this survey is based on a validated food frequency questionnaire for children or one that was especially generated. Nor do they explain why they chose to eschew the more usual tool of principal-components analysis to generate familiar dietary patterns from the food groups: e.g., ‘Western’ or ‘prudent’ with degrees in-between.^[2] Although the authors describe 4 categories, they use 5 in the graph and elsewhere; the ‘missing’ category here is Very Poor.

Beyond that, the authors do not give the reader an example of the scoring system or the weights applied to different foods, food groups, or methods of food preparation. The reader has no idea if the children with the very good diet are a subset who, e.g.: eat nuts or seeds every day (1%); eat dark green leafy vegetables every day; eat 5+ portions of fruit a day (15%); eat oily fish 3 or more times a week (6%); and never/rarely eat sugary snacks, fried food, takeaways etc. for which there are negative correlations in the data. The food frequency %s are given in the brackets and drawn from the food frequency table in section 8 (pg. 21).

This is rather important because although the authors frequently emphasise the 10,222 participants in the survey, they have SAT scores for only 3139 (pg. 6). Depending on how restrictive the criteria are for the Very Good diet, this category may be reporting the results for anywhere between 6^[3] and 628 children. If the number is towards the lower end of the spectrum, it is not infeasible that those children are from the same family group^[4] or a small number of families that would account for the dietary recommendations.

It is possible that recommendations for an ideal diet for children in the UK is based on data from a small number of children and the dietary habits of a very small number of families. If this is an accurate depiction of the data that underpin these recommendations, then this is somewhat at odds with the claims for robustness in section 3.5.

The data generated by the 10,222 respondents gave a good range of regional representation, covered male and females and provided a good sample across four age ranges. While there were some variations in overall scores…they were not marked and this report therefore focuses on the total sample. This gives strong statistical significance to the findings from a robust sample size. [pg. 8; emphasis added.]

Holford Watch disagrees for many, many reasons. Holford and Fobbester have acknowledged in some places that the survey was a self-selected sample and the data are not weighted, so the results can not have “strong statistical significance” despite their optimism otherwise. E.g.:

This very large self-selected sample delivers powerful statistical correlations between diet, lifestyle, behaviour and academic performance. This sample, while providing a broad cross-section of the public, were not randomly selected nor has the data been weighted to attempt to make the sample representative of any population. The average SAT scores of the sample were considerably below the national average. It is therefore likely that the respondents were more likely to have children with poor behaviour and academic performance. [pg. 7; emphasis added.]

There is no indication that this survey is sufficiently powered to detect the effect of food groups, food items or food preparation methods on health, behaviour, academic performance or other metrics. We even have problems interpreting some of the more straightforward data. The relative numbers of children in each category might be very important if there has been some weighting calculation to adjust the SAT scores in 4.2.1. It is readily apparent that the reported means of 2.93, 3.02, 2.94, 3.01 and 3.26 yield an average SAT score of 3.03 rather than the 2.97 that the authors report. Unfortunately, the reader has no information about the weighting. The data are not presented with error bars on the graphs, nor with confidence intervals for their p significance values.

When graphical illustrations of data are used well, they make the data easier to understand. They can present large amounts of data concisely in an easy-to-interpret graphical summary. An appropriate representation can highlight and clarify relationships that might otherwise be difficult to detect. If there are reliable values and confidence intervals then this can support extrapolation for various claims. However, this is not an example of a graph that was well designed. This graph does not condense the data in a meaningful way yet large claims are made for the relationships that it is claimed to eludicate. In its present form, this graph represents a very partial and unbalanced summary of the data because it does not indicate the number of children in each group, nor does it provide error bars. As such, this graph lends itself to misinterpretation, over-interpretation and erroneous conclusions.

We shall explore the other tables, charts and graphs in this report but we are not optimistic that they are of greater relevance or quality. Not to add to the general pessimism, but in concentrating on the literature and data analysis, we have not even touched on the methodological problems of this report. The general reader must wonder what the Scientific Advisers were thinking when they approved this report.

Notes

[1] You might also note that although the graph represents the SAT score of the Very Good diet children as if it is 4x larger than the score of the children with the Very Poor diet, there is a difference of only 0.32 points. The authors chose a very peculiar scale to express the group differences.
[2] We shall expand this more fully in a later post.
[3] We chose 6 rather arbitrarily but the table of food frequencies shows that 1% of children eat a serving or more of nuts and seeds per day. 1% of 3139 is 31 children. Maybe all of these children eat dark green leafy vegetables every day or maybe only half of them do, say 15 children. After other restrictions are applied, the number of children with a Very Good diet might be a very small number.
[4] Ceec has pointed out in the comments that the usual way to deal with data from the same family would be to correct for clustering. If the authors did this, they don’t mention it. However, given the uncertainty about the numbers in the groups, I think that it is unlikely.

Further Reading

Food for the Brain Child Survey 2007: The Promotion
Holford Watch looks at the literature review:
Food for the Brain Child Survey 2007: Review Part 1
Food for the Brain Child Survey 2007: Review Part 2
Food for the Brain Child Survey 2007: Review Part 3
Food for the Brain Child Survey 2007: Review Part 4
Food for the Brain Child Survey 2007: Review Part 5

Holford Watch appeals for help to Professor Holford and two members of the Scientific Advisory Board who approved this report and then looks at the data and analyses:
Food for the Brain Child Survey 2007: Review Part 7
Food for the Brain Child Survey 2007: Review Part 8
Why Don’t Food for the Brain Report Their Survey Results on Supplement Pills Survey: Review Part 9
Food for the Brain Child Survey 2007: Review Part 10