If I didn’t know better, I might suspect that Durham’s fish oil trial initiative has been designed as an example of the dangers of releasing scientific results via the mainstream media. I have been looking over the Durham figures, and the situation is so bad that there is even some confusion as to how many students were eligible to start on the trial and how many took the pills.
Before the initiative, Durham put out a slightly confusing press release about numbers: the sub-headline states that “County recruits 5,000 Year 11 pupils for unique study”, but the text below made clear that they hoped to sign up 5,000 of the “Year 11 pupils at Durham County Council’s 36 comprehensive schools”. It’s not clear whether Durham had a total of 5,000 eligible pupils in its schools, or the 5,000 they were hoping to recruit is just a percentage of the total.
Back in March 2008, Ben Goldacre reported that, in Durham:
The idea was to give 3m pills to 2,000 children over eight months and see if their GCSE results improved
Sounds nice and clear, except Durham are now reporting that
Initially, just over 3,000 Year 11 pupils began the study, taking the Omega-3 tablets at school and at home.
Durham also helpfully supplied a list of the schools that took part in the initiative: we were interested to see that 37 out of what Durham previously described as 36 eligible schools participated. Of course, things change when you’re doing research – but in the absence of properly reported results, it is how to know how to interpret these figures (for example, was the initial pool of eligible participants more than 5,000 students, given the ‘extra’ school?)
I would like this information for a number of reasons. One that stands out – to me – is the issue of how few pupils took up the free pills and complied with taking them. Assuming that only the 5,000 reported pupils were eligible to start on the initiative (it may well be more), only 17% of them (832) went on to take the pills and achieve a reported 80% compliance. Given that this initiative involved free pills and massive media coverage, and the children were given the pills while they were in school, this is strikingly low. While it is hard to work out what is going on with these figures, it does seem clear that these take-up and compliance rates do not bode well for any plans to roll out such schemes more widely.
Durham’s fish oil initiative thus serves as an excellent example of the dangers of science by press release. Having spent longer thinking about this than I should have done, I still cannot even be sure of very basic information – such as how many students were eligible to sign up for the study, and how many did sign up. In such a situation, I cannot see any hope of drawing reliable conclusions from the Durham (non)trial.
I hope that someone at Durham will explain to the thousands of pupils and parents and the numerous teachers who allowed their time, energy and bodies to be used in the course of the study why this research has been handled so poorly. Such an apology might be an appropriate use of a press release.
“I still cannot even be sure of very basic information…”
I’m not surprised. I can’t find a single piece of information about this botched trial that Durham County Council has released that can be relied on – it’s almost as if they are ‘throwing up chaff’ in a bid to make the whole thing as confusing as possible. They are refusing to allow themselves to be pinned down, and are managing to avoid this at the moment by providing at least two answers for each question.
I’ve been banging on about the contradictory nature of statements from figures involved in this and someone referred to the Copenhagen interpretation of quantum theory and Schrodinger’s thought experiment. As Schrodinger’s cat was both dead and alive, so this was a trial and yet also was not a trial. The number of participants was 5000, 3000 and 2000. The results are being published and are not being published. Durham CC both always and never meant to analyse the data. The measurement was actual versus predicted grades – except when it was the performance of pupils who took the pills versus the performance of those who didn’t.*
*Which, incidentally, reminded me of the Taubes piece in the NYT (that has been linked to on this blog before) about the Bias of Compliance.
Yes – the effects of focusing just on the more compliant students (or, to be more exact, 629/832 of the more compliant students) could well be one of many serious problems with this (non)trial. It’s hard to even be sure of this, though: for example, if the matched pairs compared, say, students with 78% compliance with students with 82% compliance, I doubt that the bias of compliance would be a big issue (although there are, of course, plenty of other reasons why this would be a shockingly bad idea).
The (non)trial has been so badly reported that one can’t even properly assess the problems with it.