Holford Watch: Patrick Holford, nutritionism and bad science

And please let Holford Watch read some decent research findings before they explode

Patrick Holford has been trailing the outcome from the Food for the Brain Chineham Primary School project with customary restraint; words like amazing compete with dramatic. It’s almost as if Holford’s PR company is tagging the story with the trigger words favoured by the People’s Medical Journal.

FFTB has released more details in advance of tonight’s programme but they are a mash-up of summary and numbers that fails to inform. We can say that none of the data relates to any of the issues that we listed yesterday. Maybe we lack the right Babelfish.

Holford Watch is a little uncomfortable that we are having to refer to newspapers and television programmes to discover details of the study that should be readily available in the public domain. We would like to know how many children were involved in the study so that we can interpret the results. A newspaper story reports:

The 100 pupils, aged four to 11, are being monitored through regular wellbeing surveys, and a focus group of 10 children have blood, urine and hair tests to monitor their nutritional levels.

This is in-line with the school roll if it includes both the pre-school and the special unit; however, it is especially difficult to obtain ethical consent to work with either very young children or those with special educational needs so it will be interesting to see if any children from either of these groups are in the focus group. From comments on the FFTB website, it doesn’t seem as if all of the parents co-operated but there is no news about how many children this affects and whether it has reduced the number of parents’ assessments that were available for the analysis.

It would have been tremendously helpful to the general reader if the report allowed us to place its findings in context by giving an overview of related literature. E.g., the outcome of other studies that have used a combination of supplements with children, such as Effect of supplementation with polyunsaturated fatty acids and micronutrients on learning and behavior problems associated with child ADHD (readable discussion and summary) or in a more general population of schoolchildren. It would be especially interesting to compare the results with studies that relied upon dietary and/or fitness interventions without supplementation in order to have some point of reference for the relative contributions to the results.

Interpreting the results would also be easier if we were to be given an overview of the impact of other interventions, such as the Home Reading Partnership which seems to be running concurrently with the FFTB project at the school and the extra funding that they received to help them work towards a National Healthy School Award.

One key activity which forms part of this approach at Chineham Park, and which helps us to meet the extended schools agenda criteria, is the breakfast club we run. We offer early morning group reading sessions for our children and parents as well as breakfast itself. This ensures that the children start their school day in the right frame of mind and engage more naturally in reading and other activities.

This is an excellent initiative that might, by itself, have substantially increased the attendance at breakfast club independently of FFTB (Holford claims credit for the striking increase in attendance at Breakfast Club from 8%-40%).

Similarly, the headmistress and school are to be commended for almost doubling the SATS points in the academic year prior to FFTB’s involvement, so it would be helpful to know what further improvements in the scores had been projected, even without the involvement of FFTB. [Edit: 20:30 post broad-cast: it is a little annoying that the voiceover commentary for the programme mentioned that the school SAT scores were poor last year without mentioning that they were almost double the score attained in the previous academic year. Nor did they mention that the school had already moved from near the bottom of the SAT School League Tables to up out of the bottom 200.]

Whilst SAT scores improved in 2006 participation in the Food for the Brain Project has helped the school to make further improvements in all core subjects. The proportion of pupils achieving the government’s target score of 4+ in English is up from 71% to 86%, from 50% to 71% in Maths and from 50% to 64% in Science.

I expect that the SAT scores increased so dramatically in the 2005-6 academic year because Chineham Park Primary participated in a “working with business to build readers” scheme which seems to have been very helpful. 15 staff from a local business branch donated 1 hr per week to listen to the children, amongst other donations of staff expertise.

Chineham Park Primary has become the first primary school in North Hampshire to have a sponsor governor. The company BNP Paribas (Basingstoke branch) have committed to supporting our school with 15 personnel for 1 hr per week to listen to children reading. They have financed the CRB checks, I have delivered training on ‘listening to children read’ and teamed up the company personnel to class groupings. We have also gained an experienced ‘human resource’ person from BNP Paribas onto our Governing Body.

Somebody should give that headmistress an award. Other issues aside, that reading scheme should have created stronger readers for the school to take into the academic year 2006-7; it is likely that this would make a substantial contribution to better test results.

If I seem to be lost in admiration of the headmistress and teachers, I am. Even in a comparatively small school, it was noticeable that the headmistress addressed each child by name and had an excellent demeanour with them. I should declare that many of my relatives are teachers. Many people have experienced the difference that a good, committed teacher can make. Some time ago, my brother managed to put some pupils through GCSE French and they all passed with good grades. That might not seem too remarkable, but at that time, that inner-city school had never entered any of its pupils for GCSE in the subject. It is with some degree of partiality that I add that two of my aunts are former headmistresses who had a tremendous impact on the success of their students.

The FFTB project used the popular Conners’ Rating Scales to track the children’s performance. It would be very helpful if FFTB were to clarify which version of the Conners’ Rating Scales was administered (the long or short) and whether there were any cross-cultural issues (not only when comparing the children’s performance relative to the normative north american database but for parents’ response to the forms; the school has a minority ethnic score of 17 (pdf)). It would be good to see a discussion of correlations and differences between the parents’ and teachers’ rating scores similar to the usual analysis given in a published paper; e.g., Factor Analysis and Validity of the Conners Parent and Teacher Rating Scales in Childhood Cancer Survivors and How Informative Are Parent Reports of Attention-Deficit/Hyperactivity Disorder Symtpoms for Assessing Outcome in Long-Acting Treatments? A Pooled Analysis of Parents’ and Teachers’ reports.

The general reader would benefit from an evaluation of interpreting the ratings scores in the presence of co-morbid conditions or specific educational issues. These conditions seem common among Chineham pupils because educational psychologist Melanie Herff mentions that the school has double the usual percentage of children with special educational needs. It would also be good to be given some sense of the normal fluctuation in children’s performance over a comparable time-period and at different times of the year, in accordance with (say) back-to-school viruses, colds and flu in September (which might raise the scores) or seasonal allergic rhinitis (less likely as the final post-test seems to have been performed in May). It is not implausible that ratings instruments such as this consistently report higher scores at the start of the school-year after the summer vacation; it would be good to see a discussion of this.

The Chineham Park Primary-FFTB results are interesting and are as follows (I have re-ordered elements to put the graphs in a more useful place). I would also point out that the scale-range on the y-axes differ for the graphs and over-emphasise the differences in the parents’ ratings (parents range 42-58 and teachers 44-60).

Parent Conners ratings demonstrate how pupils at Chineham Park Primary School in Basingstoke, Hampshire have made impressive improvements at a high level of statistical significance (p [less than] 0.001) in a number of measures including:

• Lack of concentration and inattention (12% improvement)
• ADHD symptoms (13% improvement)
• Hyperactivity and impulsiveness (13% improvement)

Teachers’ Conners ratings also showed improvements in similar scales, notably at a high level of statistical significance (p [less than] 0.001) in:

• Lack of concentration and inattention (9% improvement)
• ADHD symptoms (9% improvement)
• Hyperactivity and impulsiveness (9% improvement)

While these results occurred for the school as a whole, children who had initial ratings above 60 at the outset, indicating learning, attention and behaviour difficulties, showed greater degrees of improvement.

It would much easier to interpret the numbers from the raw data than the graphs not least because that data should include standard deviations (SDs), which these graphs do not illustrate and are very important when interpeting the results. A scatter-plot with SDs might have been a much clearer way of presenting the results for a general audience; it might even have been interesting to see an overlay of the parents and teachers scores to assess correlations and differences between scores and trends.

It is notable that the graphs make some of the differences in scores between baseline and final post-test appear to be substantially larger than they are. It is also difficult to use them to verify the reported % improvements. When Holford Watch ran some quick calculations, some of the figures were out by 1-2%, both improving and worsening some of the reported improvements. E.g., the parents’ score for concentration and inattention is nearer to 11% (10.71) than 12% and the teachers’ for the same is near to 10% (10.3) than 9%; the teachers’ hyperactivity score looks like it is closer to 8% than the reported 9%. Although Holford is notable for dodgy arithmetic and poor reporting of numbers, we have to assume that it is the graph that is at fault (a table of the data would have been invaluable).

The general reader can’t assess the claims about the subset of pupils with baseline scores above 60 because they are not represented on the graphs, nor are there SDs. If the reader knew the SDs it would make it easier to assess the claims of improvement. Similarly, it would help if we knew the numbers of students for whom all of the parents’ and teachers’ rating scores are available.

Sadly, there is no point in relying on FFTB’s experts to provide a helpful commentary. Holford Watch will comment further on the Woodhouse summary in another post. Suffice it to say that the sub-group analysis is very confusing and not at all enlightening.

All in all, it is difficult to understand the context for Patrick Holford’s exultant conclusion (particularly in the absence of a decent literature review):

The scale of these results could not been achieved by diet alone. This project suggests that an improved diet, plus supplements and daily exercise might be a winning formula combined with good quality teaching and an exciting curriculum. Our next task is to find out exactly what is making the difference – is it the diet, the supplements, the exercise, the change in awareness or all of it.

However, the key is involving parents and children every step of the way to take on board a healthier lifestyle. Not all parents participated but many did make positive changes. We hope that transformations like this will make the Government sit up and listen and provide the necessary funding for further research and support the Food for the Brain project in other schools in Britain. [My emphasis.]

It sounds good on television and it reads well but is there any evidence that the changes could not have been achieved by diet and the other interventions without supplements? Is this claim more than truthiness, a thing that Holford feels to be true or is it evidence-based?