Holford Watch: Patrick Holford, nutritionism and bad science

You may remember that we contacted Professor Patrick Holford on January 11 to ask him some questions about the Food for the Brain Child Survey 2007. We did not receive any response. Enquiries to Professors Philip Cowen and David Smith from the FFTB’s Scientific Advisory Board did not provide any useful response to questions about the report but did produce some useful comments.

We still do not have a direct response but they have just posted some Q&A about this report.

Disappointingly, in their response to some simple questions, FFTB has already erred in simple arithmetic, reporting only 3130 children in the SAT group rather than the 3139 mentioned in the report (pg. 6). The report states that 1% of the children eat nuts and seeds once a day or more but when asked for the raw number, they quote 92; this is not 1% of either the full sample of 10,222 (which should be 102) nor the SAT group of 3139 children (it matters because of their claims). And the claims for statistical significance go awry but more at another time.

We are going to accept the FFTB invitation to pose questions about the report. You may recall that we wrote more than 10 posts, discussing problems with the literature overview and the data analyses. We shall distill those down into more questions. For now, we shall just resubmit most of the ones that we asked in January. We have produced them below. If FFTB has answered some form of the question in their current Q&A then we comment on what they provided by way of a response (it would be wholly inappropriate to style some of them as answers – you will understand why). We have put the FFTB responses in green/bold (it might depend on your browser).

Email sent from Holford Watch to Patrick Holford January 11 2008

We’re also in the process of analysing the science of the September 2007 Food for the Brain Child Survey, and it would be very helpful if you could answer a couple of questions and respond to a couple of criticisms. If FFTB is looking to publish this as a journal article, I suspect that these points will be among those raised by reviewers anyway – so hopefully shouldn’t create too much additional work:

In most of the report, it’s not clear whether the nutritional data are drawn from the data collected on all 10,222 children or just those for whom their SAT scores are known (the 3,139). 3.5 states that the data for assessing food/food group associations were generated from the 10,222 respondents. However, it is unclear whether this data were comparable to those of the sub-group of the 3,139 for whom there were SAT scores or not; it is this latter group that is represented in later analyses. Could you clarify this?

HW: We could comment further about this but any pretensions of relevance for these data are completely undercut by the later admissions that their recommendations for an ideal diet for UK children rests upon a comparison involving the outliers of their survey who make up 0.72% of the whole.

The graph 4.2.1 is represented as showing a “direct and consistent increase in SAT scores with improving diet”, yet it doesn’t. The improved diet is based on a score that is derived from the survey results of these children with a substantially below average SAT score. It is difficult to tell from the data but it seems as if the no. of children in the ‘very good diet’ category may be as low as 120 or so – is this the case?
FFTB responds: The difference between the the SAT scores of those with the very poor diets and those with the very good diet was 11% (p<0.05).This data suggests a trend towards increasing SAT scores with improving diet, not a direct and consistence [sic] increase. We accept this criticism.

Holfordwatch put up an explanation of what is wrong about this graph and FFTB’s claims. FFTB’s response is inadequate. They have provided SDs but no confidence intervals. There is a startling revelation that the FFTB dietary recommendations are grounded in a comparison of two outlier groups, as represents the very good and very poor diets, with sample sizes of 42 and 32 respectively. They really don’t seem to notice that this adds up to a grand total of 0.72% of the whole survey sample or 2.36% of the SATs sub-group. At these levels, and with an acknowledged unrepresentative sample, and no awareness that some of those children in those groups may be from the same families, and might need further adjustment for clustering, you are more likely to be discussing statistical noise with those outlier groups. Plus, you really don’t claim “statistical significance” as FFTB does when mentioning these figures in response to a later question – certainly not without divulging your confidence intervals. We should also add, that somewhere, they have lost 9 children; the numbers that they give add up to 3130 rather than 3139 (FFTB report, Very Good 42; Good 868; Neutral 1706; Poor 482; Very Poor 32). Normally, we wouldn’t draw much attention to this but, unfortunately, given the small numbers in the outlier groups, these missing children might be disproportionately important.

Our problem with table 4.3 is that we don’t understand if the data in the first 3 columns are drawn from the analysis of the 10,000+ or just the 3,000+ that are represented in the SAT scores of column 4. Beyond that, we don’t understand what is meant by the effect size mentioned in the notes. The report does not mention how FFTB calculated the effect size or any useful information such as SDs. We have never seen an equivalence drawn between a % and the effect size: an effect size of 8% is said to indicate that a high consumer of a food has an 8% higher score than a low/non consumer. It also seems quite novel to calculate an effect size for an observational survey without any intervention?

The wholly inadequate response to a simpler form of this question is as follows, however, it can be summed up as, “We didn’t mean effect size, qua effect size, or any of the normal formula that involve knowing SDs etc.”.

FFTB: The ‘effect sizes’ were calculated by looking at the prevalence of respondents in each (high or low) consumption band giving a ‘very good’ rating. The increase or decrease in prevalence between the two represents the ‘effect size’. This represents the increase (or decrease) in likelihood of a very good rating. The comparisons were only made where the variances in prevalence were statistically significant (shown by the colour bandings). We have not calculated standard deviations for these values. We accept that this grid may be trying to ‘do too much’ and we may have compromised on clarity by perhaps trying to be too concise. We are looking at a simpler mechanism that will more clearly show the dramatic difference between consumption groups and the apparent effect of increasing consumption of different foods on stated overall health, stated academic behaviour, SAT scores and a series of everyday symptoms.
HW current response: We think your data are irredeemable. Trying to be concise was not your over-riding problem: we have commented on this in several places, notably in Food for the Brain: Child Survey: Review Part 10. The way in which the authors calculate the impact of foodstuffs on behaviour and academic performance is beyond the normal understanding of “That’s wrong”.

4.4 has ideal diet recommendations for children. The recommendations are a little tricky to interpret; do the 5+ fruit and 5+ vegetables include the portion of nuts and seeds and the dark green veg or are these last two in addition?

We don’t have any helpful gender breakdowns. Given the better academic performance reports for girls than boys (6.3), it is a little odd that we don’t see analysis of different SAT scores, particularly the higher ones that are drawn from comparatively small numbers that may be disproportionately female?
FFTB respond to a simpler form of this question: There were no statistically significant gender variances in the SAT score data. The data in 6.3 represents parental subjective view of academic performance and this may (or may not) include other subjective views of academic performance aside from SAT scores – (see para 3.5 in the report that confirms this and explains use of total sample for this high level report).

HW current response: This really isn’t good enough. It is clear that girls have better reports than boys; given the tiny numbers in the outlying groups it is feasible that there are gender differences. FFTB should release the raw numbers. The reference to the report really doesn’t explain anything. FFTB report says:

girls are more likely than boys to perform well academically (Girls 26% very good versus 16% of boys). [HW has not evaluated this claim: section 6.3, pg. 17]

Those completing the FFTB questionnaire were asked whether the children used essential fat and multivitamin supplements, but the results of this are not included in the report. Was there a particular reason for this omission, and are these results available?

HW comments more fully on the issue of these missing data about supplements.

7.3 reports improved academic performance of children who eat a portion of nuts/seeds per day. However, in the table for 8 we discover that 1% of the sample achieved this. Is this 1% derived from the 10,000+ sample or the 3000+ set.
FFTB respond to a different form of this question: 92 children reported eating nuts and seeds every day. The ‘3-5’ and ‘at least once a day’consumption groups did not show a statistically significant effect on SAT scores in this case. The only statistically significant finding was reported, stating “regular consumption of fresh, raw nuts and seeds is associated with a statistically significant (p<0.001) 7% increase in mean SAT scores at a consumption level of 2-3 times per week.” This trend is consistent with those found in relation to parents rating of behaviour and academic performance.

HW responds: We have many problems with this that we shall elaborate on at another time. For now, ask yourself whether it is accurate for 92 children to be 1% of either 10,222 (the total sample) or 3139 (the SAT sub-group) children; small numbers can mean a lot here. Further, if you consult 9:16 on pg 30, you will see that there is a SAT score difference of the size that FFTB otherwise rhapsodises about. A score of 3.14 at a consumption frequency of 2-3 per week but 2.98 at once or more a day (2.96 for 3-5 time a week and 3.03 for rarely). FFTB bruits about there there is a significant 7% increase in the mean SAT scores. Actually it is 6.8%; we don’t mention this to be persnickety, but because you can obtain a 6% difference between the mean SAT score of the 2-3 per week and the 3-5 per week, a score that falls with increasing consumption.

There is something badly amiss with these data and their analysis. Of course, if there are errors in the size of the groups, there is no possibility of these data being correct. We were going to digress on to more reasons why these data can not be correct but it would really take far too long here.

For the whole of 9, we are not clear whether the data are from the 10,000+ sample or just the 3000+ SAT set – perhaps you could clarify. It also seems a little odd to put the 2 lowest groups together as one class and contrast them with a small group of the comparatively well-behaved or high academic performers: what was the rationale for this?

Re: nuts/seeds, how does 9:16 relate to the recommendation for a daily serving?

As above, but the drop in SAT scores would seem to mitigate the usefulness of this recommendation. If you accept these data (which we don’t) your SAT scores fall as you increase your consumption; however, this looks like statistical noise to us.

10.1 is interesting because it would have been helpful if they had indicated how many children fell into each group so that we might have some idea of the samples. It seems as if the ‘very good diet’ group would have to be part of the subset of children who are in the 6% who eat oily fish 3+ times per week, 2% who frequently eat dark green veg, and the report refers to 1% who eat nuts/seeds daily: is this correct? What is n for this group – it looks like it might be very small?
The sample sizes are as follows. [FFTB report, Very Good 42; Good 868; Neutral 1706; Poor 482; Very Poor 32].
While these are small sample sizes the difference was statistically significant.

You will notice that somehow they have lost 9 of the 3139 children. Unfortunately, given the small numbers in the outlier groups, these missing children might be disproportionately important. You will notice that they are relying upon a tiny number of children to estimate their ideal diet recommendations for children throughout the UK. It is hard to understand why they think these outliers offer “statistically significant” results.

We asked the above questions before we fully understood just how flawed the FFTB Child Survey 2007 is. We will provide further questions to the FFTB and their SAB and post them here and notify you of any responses.

Further Reading

Food for the Brain Child Survey 2007: The Promotion
Holford Watch looks at the literature review:
Food for the Brain Child Survey 2007: Review Part 1
Food for the Brain Child Survey 2007: Review Part 2
Food for the Brain Child Survey 2007: Review Part 3
Food for the Brain Child Survey 2007: Review Part 4
Food for the Brain Child Survey 2007: Review Part 5

Holford Watch appeals for help to Professor Holford and two members of the Scientific Advisory Board who approved this report and then looks at the data and analyses:
Food for the Brain Child Survey 2007: Review Part 7
Food for the Brain Child Survey 2007: Review Part 8
Why Don’t Food for the Brain Report Their Survey Results on Supplement Pills Survey: Review Part 9
Food for the Brain Child Survey 2007: Review Part 10