Durham (non)trial fails to show any benefit to fish oil (Equazen EyeQ) supplementation

Yesterday, a draft of Durham’s report on their fish oil (non)trial flopped into my inbox like a fortnight old fish – courtesy of a Freedom of Information Act request, and Durham’s helpful FOI coordinator.  Sadly, the report stinks to high heaven, but there are a number of things to note.

Firstly, the report is quite clear (page 2) that the trial has failed to demonstrate any benefit from the supplement pills.  Given the lack of a placebo control group, the benefits that attach to more compliant children and families, the raised expectations due to media coverage and so forth, concerns about what they chose to judge ‘success’ by, other factors like improving school food, etc., the performance ‘benefit’ of those on the supplement group is actually rather disappointing.  It’s a shame that Durham’s recent press release on the (non)trial did not make this clear.  When Durham does publish this report (in the prestigious journal of the Durham Council website – clearly, they had good reason to withhold the non/trial data in order to avoid prejudicing publication) can I suggest something along the lines of ‘Durham trial fails to demonstrate any benefit from EyeQ fish oil pills’ as the headline of the press release announcing publication?

Anyway, having noted that headline point, I will now go through this awful report in more detail. First, I should note that this is a draft: this draft has been passed to Durham’s web team for upload to their site, but I guess it is still possible that parts of this will change (if it were me, I would start from scratch and look to rewrite entirely).  However, as this report is likely to crop up in the media once it is made available, I will go through the draft here.

Durham have managed to produce a 3 page report of an uncontrolled (non)trial with a high drop out rate; the report fails to give adequate detail of methods, analyses applied or results.  The report therefore fails on numerous levels.

The report starts by saying that the initiative involved omega-3 supplementation.  However, Equazen EyeQ tablets contain evening primrose (omega 6) as well as fish oil, and it is not clear why this is not mentioned.  If the sciency write up of a (non)trial fails to even mention the ‘active’ ingredients in the pills used, this is a bad start.

The report then points out that “consumption [of the pills] was either in school weekdays and home weekend and holidays or all at home”.  Nice to see that they had a consistent system for distributing pills and monitoring compliance in place, then; of course, only cynics like me would think that compliance (and over-reporting thereof) might be more of an issue for those taking all the pills at home.

The report then notes – in passing – that “In addition many settings developed ‘healthy eating schemes’”.  While I’m all in favour of such schemes, they could clearly also impact on children’s health and even exam results and thus – in the absence of randomisation, a placebo control group etc. – add further significant bias to the (non)trial’s results.  Are children who comply with a supplement regime also more likely to comply with a healthier diet? By the way, I’m only on paragraph 3 of the report – not a good start.

Still on p. 1, the report notes that they compared ‘matched pairs’ of children.  They tried to match those who took the pills and those who did not, matching school, gender, prior attainment and social background (although concepts like social background are complex, and the report does not say how this was matched).  The report clearly fails to match compliance, and acknowledges that

participants were self selective and our matching process indicated that those taking part in the initiative tended to come from more affluent areas of the county and the difference was statistically significant.

On page 2, the report claims statistically significant results, while also stating that “this was not a scientific test but we wanted any young person who was interested to be able to have the opportunity to access free omega-3″.  The report goes on to state that

we make no claims that these results can be attributed only to omega-3 supplementation.  The benefit may be a placebo effect, or other factors may be responsible for the differences noted…it may be that students who achieved 80%+ compliance are particular [sic] well organised and have families who provide support at home – so they may have done better anyway.

Quite right.  That is what happens when you organise a trial really badly – you don’t get any meaningful results.  If I had participated in this trial – letting the organisers use my time and body in the hope of generating useful research – I would be annoyed.

Page 3 primarily discusses issues around compliance – some settings had a 90% drop out rate.  What is covered is mostly commonsense stuff, like the fact that compliance may be better if children (not just parents) want to go on the trial.  All fair enough – although I would like to see more critical analysis of the compliance data, rather than a list of commonsense statements about compliance which may or may not be supported by these data.  It would also be good to see some discussion of the broader issues raised by low-ish take-up and high drop out rates: even if EyeQ pills are useful for ‘typical’ children, it looks like it will be hard to get all that many people to take them consistently.  And would it be easier to feed children oily fish on a regular basis, instead?

I am also worried by the fact that there is no discussion of adverse effects.  Were these monitored?  If so, why aren’t they discussed in the report?  And, if not, why not: is it ethical to test pills on children without monitoring adverse effects?  I also note (p. 2) that children “in special schools where the chosen measures were inappropriate” were excluded for the analysis: were the children and parents told about this, before embarking on the trial?

It’s worth ending by highlighting some features of and deficiencies in this report (or though we may return at a later date to highlight a few additional issues).  Firstly, the report fails to demonstrate any benefit which is attributable to the active ingredients of the supplement pills used – the report admits this, and the appalling trial design meant that this was always going to be the case.  Given the lack of a placebo control group, the use of media etc. to raise expectations, the likely benefits attaching to more compliant pupils and families, etc., the performance ‘gains’ in those taking the supplements are actually quite disappointing.  Secondly, as I have shown above, the report itself is of an extremely poor quality – reflecting the abysmal design of the trial on which it was based.

Thirdly, Equazen and Durham have both gained extremely widespread positive media coverage based on this fiasco.  Good quality research takes a little effort, but is not much more expensive than bad research – but, frankly, given the many ways in which this awful trial was able to succeed, I can see why many would choose not to bother.

Finally, did I mention that we’re not talking lab rats here – this was a trial on children.  The successes of Durham and Equazen in promoting this media event came at the expense of a poor-quality trial – which was carried out on a large-scale, and using children (in some cases, I suspect, vulnerable children) as the subjects.  It’s not just fish oil pills that can cause nausea.