Professor Patrick Holford of Teesside University and Head of Science and Education at Biocare enjoys his own special reality. You may recall that in The Great Gatsby, Fitzgerald describes newspaper reports of his hero’s murder as “grotesque, circumstantial, eager and untrue”. HolfordWatch feels the same way when reading some of Holford’s more outrageous interpretations of scientific literature.
Is “outrageous” a bit strong? No. Holford repeatedly demonstrates breathtaking ignorance of statistics yet has an unswerving belief in his own ability to interpret those of scientists and researchers who do understand statistical techniques and tools. Morever, he represents his mis-information as expertise to the surprisingly gullible lifestyle journalists who, too frequently, cover stories that are best left to journalists with some understanding of science or medicine. And, not least, he promotes himself as an expert to the public and sows disinformation.
It is no wonder that the statistical analyses in the Food for the Brain Child Survey 2007 are as poor and worthless as they are.
We offer you one straightforward example where Holford not only mis-interprets simple summary statistics but branches out to claim:
It really makes me question the integrity of the authors and the journal. Let’s explore that for a minute with a ‘conspiracy theory’ hat on. Last week pharmaceutical drug sales topped $600 billion. The number one best seller was Lipitor, a stin [sic] drug for lowering cholesterol… [Don't be fooled by the Omega 3 scam]
Just what is it that has provoked Holford’s faux theatrics and lead him to suggest the donning of tinfoil hats? It is his interpretation of summary data from Hooper et al’s systematic review of Omega 3 for mortality, cardiovascular disease and cancer.
It’s my job to read and analyse the whole paper and it soon becomes clear there is something very fishy going on…
[T]he authors…analyse all the studies using fish derived sources of omega 3. They include twelve studies, nine of which show a benefit, one of which shows no effect and two show a very small negative effect. However, combined, in the way that this review analyses the data, this seemingly obvious beneficial effect doesn’t come up as significant.
What I find particularly deceptive I that this obvious skew is not even discussed in the research paper. It really makes me question the integrity of the authors and the journal…
[Don't be fooled by the Omega 3 scam]
Well, if that is all that is underlying Holford’s concerns then maybe he should start planning a very classy apology to the authors of that systematic review and the BMJ. I’m not sure what scale of apology is appropriate when you have questioned the integrity of the authors on such insubstantial grounds but you might have your own opinions on that.
The reason that “this obvious skew is not even discussed in the research paper” is because it doesn’t exist. The fact that it doesn’t exist is apparent within half a second to anybody who actually understands how to interpret a forest plot (aka confidence interval plot or blobbogram). If you already know how to do this, please skip the next section, if not, please bear with us for a couple of minutes which is all it will take for you to understand this unbelievably simple form of data summary.
the blobbogram at a glance
You have systematically sorted through your records and other relevant sources and come up with a number of trials that meet your criteria. For each of those trials, you display your findings as a blob (your measured effect or point estimate of the effect of your intervention). You draw a horizontal line that extends either side of your blob to represent the results and their confidence intervals. The longer the line, the more uncertain that result and, conversely, the shorter the line, the more certain the result. The diamond represents the combined results of all the trials.
The vertical line marks the point about which the horizontal lines would cluster if the two treatments (typically an active treatment and a control group) that are compared in the trials had similar effects: it is the marker for no effect. If a horizontal line touches (or crosses) the vertical line, then that individual trial did not report results that show a clear difference between the treatments (this is in bold because it is important later). The value of this vertical line is 1: if a horizontal line neither touches nor crosses it, then the result is individually statistically significant.
And finally, when the diamond (the combined results of all the trials) is positioned nicely to the left of the vertical line it indicates that the treatment under consideration is beneficial. A diamond that is positioned to the right, of the vertical line would indicate that the control did more harm than good or was of greater benefit than the intervention.
In the above example, it is clear to see that three blobs have horizontal lines that cross the vertical line (their results are not statistically significant) and two blobs are positioned to the left of the vertical line and their confidence intervals do not cross the vertical line (their results indicate benefit and are statistically significant).
Some gratuitous comments that you don’t need to interpret this diagram but are generally useful. You may also have noticed that line for trial 4 has a blob on the left but crosses the line. This shows that the value of the point estimate looked as if it were indicating a benefit for the intervention, but the confidence interval (interval estimate) shows that if the trial were to be repeated, then the value might lie to the right of the vertical line, or anywhere along that range: it is not statistically significant. You may also notice that trial 3 has a long line; this indicates a wide confidence interval, and, even in the point estimate had been on the left of the line, you could not have claimed any significance for the results i) because it crosses the line of no effect (the vertical) and ii) you would have misgivings because of the range of the confidence interval. (And, although the blob was to the right of the line, this is not statistically significant.)
Back to the rough guide to blobbograms, you will see above that the diamond is nicely to the left of the vertical line which confirms that, overall, the treatment depicted in our illustration was of some benefit and that this aggregate is statistically significant.
back to holford’s analysis
The following diagram differs from the above because each of the horizontal lines has the name of a trial to the left of it, and the numerical range of the confidence interval for its results to the right (this is useful when something is so close to the line that you need to check the range to see whether it includes 1, and therefore crossed the vertical line and is therefore not statistically significant). We should also mention that the size of a blob in a horizontal line relates to the relative weighting of that study. E.g., the number of participants in that trial: crudely, a large blob shows a higher number, and a smaller blob a smaller number.
We shall repeat Holford’s analysis of this diagram of the impact of marine Omega 3 on mortality (consult the BMJ’s full figure rather than excerpt to see the table legends etc; click our thumbnail to see a larger version of the relevant section of the summary data).
[The authors] include twelve studies, nine of which show a benefit, one of which shows no effect and two show a very small negative effect. However, combined, in the way that this review analyses the data, this seemingly obvious beneficial effect doesn’t come up as significant.
Is Holford’s interpretation what you see? No? It looks to us as if 10/12 of those trials either touch or cross the vertical line and their results indicate no benefit. (GISSI-P is the 7th line and looks like it might touch, but, if you consult the confidence interval on the right, you will see that it does not cross 1 or the vertical line.)
Unsurprisingly, the diamond which is the aggregate of these trials, is towards the left of the line but also crosses it which is why the authors gave their nuanced conclusion. So, Holford is mistaken: there were 2 trials (not 9) with a positive (and significant) result and the others did not show a significant benefit. The “obvious skew” is in Holford’s interpretation, not in data summary nor in Hooper et al’s nuanced interpretation of the results.
The first trial, Burr, has a wide blob (large number of participants), a short horizontal line (you can be confident in the results) and lies to the left of the vertical line; it shows a positive effect. However, the second trial, Kaul, obviously has a small number of participants (small blob) and a lengthy horizontal line that crosses the vertical line. So, even though the point estimate of the blob is to the left of the vertical line, the range of values along which the true value may lie is large which means that the results are not significant.
At a glance, you can see for yourself that Holford’s interpretation of this diagram was egregiously wrong.
As for that “combined, in the way that this review analyses the data, this seemingly obvious beneficial effect doesn’t come up as significant” remark, we have a simple observation. There is no statistical rehabilitation camp where you can send data until you can compel them to report the result that you want. You can analyse those data any way that you like and, as long as you are paying due care and attention to the meaning of significance and confidence intervals, you will never get a different result. You can not distort results that are not significant to make them so. For those who have found that the little excursion into statistics has sapped their concentration and would prefer an aphorism, you can not make a silk purse out of a sow’s ear.
Once more, we refer you to the fine paper by Kruger and Dunning that explains why so many people have no grasp for the limits of their own competence or any appreciation for the relative competence of others.[2] However, this stark competence gap is the more remarkable when one considers that one of the authors whom Holford is criticising here is Professor Carolyn Summerbell, one of his ‘colleagues’ at the University of Teesside.
Aside from the substantial apology that he owes the authors of the systematic review, Holford’s lack of understanding of this basic statistical tool raises some uncomfortable questions. Just what skills will he impart to the young minds for which he will be responsible at the University of Teesside? Will he try and persuade any students that he supervises that they should interpret a blobbogram in the same way that he does?
Update 11 April In the comments, Dr Aust points us towards von Schacky and Harris discussion of the null conclusion of the Cochrane Review. There are some commenters such as Truswell who question whether the Cochrane Review process is appropriate for assessing what are essentially observational and cohort studies. These are legitimate issues that are soberly and thoughtfully discussed. On the other hand, we have Holford, criticising the authors on the basis of his interpretation of this figure and questioning the integrity of the review authors and the journal. And then we have his explanation that this is part of a drive to protect the market share of statins.
Overall, as the authors remark, many people (themselves included) expected an unequivocal result that would indicate an attractive, statistically validated relative risk rather than the (frankly) unimpressive RR of 0.87 with a confidence interval that mandates a note to the inclusion of unity. There are some researchers such as Brignell who advocate the strong position of discounting any RR that has a point estimate of less than 2 or, if risks are reduced, be below 0.5. Others argue that unimpressive RRs are acceptable for epidemiological findings where the numbers of population involved are so large that even a less than stellar RR will benefit large numbers of people so the question is then one of whether it is justified by risk, benefits and cost.
Beyond this, of course, we have to mention John Ioannidis: Why Most Published Research Findings Are False. Ioannidis writes:
The smaller the effect sizes in a scientific field, the less likely the research findings are to be true.…[R]esearch findings are more likely true in scientific fields with large effects, such as the impact of smoking on cancer or cardiovascular disease (relative risks 3–20)…than in scientific fields where postulated effects are small…(relative risks 1.1–1.5)… [I]f the majority of true genetic or nutritional determinants of complex diseases confer relative risks less than 1.05, genetic or nutritional epidemiology would be largely utopian endeavors.
Notes
[1] Hooper et al. strongly recommend that interested parties should read the full Cochrane Review from which they extracted some of the data and findings for this systematic review. Omega 3 fatty acids for prevention and treatment of cardiovascular disease (Review) (pdf) They we had already anticipated and assessed most of the subgroupings and individual outcomes that BMJ commenters suggest in the original Cochrane review.
As stated in the paper, the authors emphasise:
The overall result from the meta-analysis was a relative risk suggesting benefit of omega 3 intake on mortality, but a confidence interval including unity (total mortality RR 0.87, 95% CI 0.73 to 1.03 with significant heterogeneity) so that we are not quite so sure.
[2] You may be interested to read an overview of the Justin Kruger and David Dunning paper that discusses how difficulties in understanding one’s own incompetence can lead to inflated self-assessments.
What’s even more amazing is that when they then shared the performance of other participants with the people who performed poorly (hoping that they would then adjust their self-perception downward) people who scored poorly failed to adjust their self-perception of their performance. In other words, they are completely unaware of their own [in]competence, and can’t detect competence in others.
It really it a very helpful paper that explains many otherwise inexplicable actions.
A Photon in the Darkness likewise offers a helpful discussion of this paper: The Arrogance of Ignorance
38 Comments
April 10, 2008 at 2:18 pm
[...] some thoughtful and spirited commentaries about the level of Holford’s scholarship that cast doubt on his ability to understand the research that he reads and cites with such abandon. Holford has written odd items about My Right to Be Called a [...]
April 10, 2008 at 5:17 pm
Do you know if Professor Patrick Holford of Teeside University, or his staff, read HolfordWatch?
I’m surprised that someone with a BSc in Experimental Psychology would apparently not know the basics of stats – imagine that type of thing is useful to budding psychologists.
I wonder if he will see fit to correct his website on the basis of HW pointing out his apparent errors.
If he does that’s excellent – but rather awkward for Prof Holford.
If he doesn’t I wonder how the more erudite employees of Teeside University will react to their visiting professors interpretation of statistics.
April 10, 2008 at 5:30 pm
I seem to recall it being pointed out (I don’t have the reference to hand but it was probably by Thomas Gilovich or Massimo Piattelli-Palmarini, who have both written accessible books on cognitive errors) that people seem to naturally feel that more and more insignificant (or even zero) results sum to more ‘significant’ evidence for the desired conclusion. The ‘no smoke without fire’ effect.
In this case, Holford’s ‘seemingly obvious beneficial effect’ will feel as if it trumps any valid statistical analysis.
April 10, 2008 at 5:49 pm
Well, no lesser a person than Professor Holford himself has informed us that he is too busy writing books and helping people to be healthier to trouble himself with accuracy (I paraphrase, slightly).
Yes, there was rather more stats in the course than the young Holford enjoyed but his degree certificate is in plain ordinary Psychology, not Experimental Psychology. I don’t know why he cleaves to the Experimental claim…
Stephanie Fox claims that she and Holford will only entertain documented claims. She is such a whizz in the publishing industry that apparently blogs don’t count and links don’t count as documentation.
It is so funny – she obviously has no awareness of Holford’s PR company’s actvities in re: the wiki disaster of 2007 or the suspicions about Hickory and whether there is another sockpuppet in play or another meat puppet…
April 10, 2008 at 5:52 pm
If Stephen Colbert does that sort of truthiness, it’s funny.
When a visiting professor at a UK university does it, it is inexpressably sad.
April 10, 2008 at 6:11 pm
“I wonder if he will see fit to correct his website on the basis of HW pointing out his apparent errors.”
They’ve amended something today actually – the title of the journal that carried the Hardman and Hart paper had been incorrectly rendered on the “Patrick’s Full Responses” webpage as ‘the Journal of Nutrition and Food Science’, rather than as ‘Nutrition and Food Science’. [The distinction is important, as the Journal of Nutrition and Food Science is a peer reviewed journal, indexed on Pubmed, whereas Nutrition and Food Science is not - which most of you were probably aware of before I was]
Which is why I was surprised to read of their stance here: “Stephanie Fox claims that she and Holford will only entertain documented claims. She is such a whizz in the publishing industry that apparently blogs don’t count and links don’t count as documentation.”
Perhaps Patrick’s errors are so numerous that his colleagues/staff simply don’t have time to correct them all? Which makes you wonder why Patrick puts so many of his thoughts in the public domain where they can be challenged.
April 10, 2008 at 7:58 pm
jdc325 – rats, do you think we are enabling him in some bizarre humilation ritual?
If you are interested in another classic that comes under the heading of “making stuff up and hanging in on the nearest hapless journal paper”, might we interest you in Holford’s arguments for chromium and cinnamon?
Scroll down and enjoy the Kahn – it really is a classic.
April 10, 2008 at 8:26 pm
Even by the standards of Professor Patrick Holford from Teesside University that really is a quite spectacular misunderstanding of basic undergraduate evidence based medicine.
April 10, 2008 at 8:35 pm
Please be careful with this. Holford is not incorrect to say “results indicate no benefit”, if one interprets this as saying that their point estimate (center of their interval) is not close to the null value.
Of course, that’s not what a meta-analysis uses; its input is the range of values each study estimated were consistent with their respective data – their confidence interval.
I don’t think Holford understands the difference – but I don’t think he’s alone in that.
April 10, 2008 at 8:55 pm
Hi anon – Holford said:
He is using his interpretation to “question the integrity of the authors and the journal…”. Do you think that his perspective is sustainable?
April 10, 2008 at 10:53 pm
Generally, I like your thoughtful analysis of misinterpretation of statistical results. However, I was a little concerned when you wrote: “You draw a horizontal line that extends either side of your blob to represent the results and their confidence intervals. The longer the line, the more uncertain that result and, conversely, the shorter the line, the more certain the result.”
Because this is somewhat vague it is hard to argue with. However, if it means, as I suspect it does, that a particular small 95% confidence interval argues for a “more certain” result, then I think that’s wrong.
The general misconception is this: a particular 95% confidence interval does not, in general, contain the sample, population mean 95% of the time re-sampling is carried out. It might contain that mean around 80% of the time, but in the case of a binomial proportion it could be as low as 20%.
The confidence interval is a random variable, and on one instance tells you anything at all about about the certainty of the result.
April 10, 2008 at 11:48 pm
Sure – that is a simplification for an over simple diagram. If you can provide a non-frightening link for interval estimates that is suitable for inclusion in a blog post I would be very pleased to pop it in.
The main point of this post is that this chap is a visiting professor who was using his interpretation of this diagram to question the integrity of the authors of the systematic review and the journal. Yet, his interpretation is flawed.
Mine is a trifle bland because I didn’t want to get frighten off the nervous by laying out all of the complexities. I have, before now, written blog posts that not only have extensive references and footnotes but further references for the footnotes. I’m trying to simplify but it is important to do that without sacrificing the necessary level of understanding to cope with the example.
I would be grateful for a better, simpler re-write of any sections that are raising alarms or “Hmms”. Or, “Yes, I see your point but would prefer this wording”.
April 11, 2008 at 12:13 am
As an example of the sort of maths one typically encounters from Holford, I present another example where he was abusing people for failing to highlight what seemed to him to be a striking finding.
Just to choose one in an area in which Holford claims particular expertise, we looked at a section in his Sept. 2007 100%health newsletter: Is Taking Vitamin C For A Cold A Waste Of Time? (pp 4-5). Holford gives an overview of the recent Cochrane Review on the topic of vitamin C and colds: pdf for whole report; plain-language summary. Holford wrote:
And yes, that would have been an amazing headline and one that he would rightly lament the newspapers and other mainstream media somehow missing. Just think of the potential productivity increases at work and in schools if children could experience “up to a month” of fewer ‘cold’ days per year. Except, the authors didn’t write any such thing. Those figures do not “equate to up to a month less ‘cold’ days per year”. Holford’s ability to interpret and communicate numbers has long ceased to be amusing and become irritating.
Take a moment to think about it. If the reduction of “up to a month” is truly a reduction of 13.6% then think of how many days a year those children would have to have a cold to make that reduction true. What does your instinct tell you on this point without even engaging your mental arithmetic skills? Do you have a vague feeling that this would imply that these unfortunate children would have to experience more than 200 days of cold symptoms a year in order to experience a 13.6% reduction that amounts to “up to a month”.
What does the primary source, the actual Cochrane Review (pdf) say?
So, that’s a pooled estimate that is extrapolated from the relevant trials; the authors refer to 28 days as the upper estimate of cold morbidity in children under the age of 12 rather than the general range, and the reduction might be 4 days from that upper estimate. Just to belabour the point, 4 days is nothing like “up to a month”.
Yet, such mathematical interpretations are being promoted by a visiting professor in a UK university and used as a basis to attack the integrity of others or as foundation for allegations of bias. And, these ‘facts’ are being circulated in a newsletter that he charges membership fees to receive and that he promotes as being ‘the truth’.
April 11, 2008 at 2:30 am
Hello dvnutrix
No, I don’t think Holford’s argument is tenable; his statement about Omega 3 being “obviously” beneficial is ridiculous. But to argue this successfully, one has to avoid over-egging things.
The description above makes a (literally) bold statement that because the meta analysis doesn’t “cross the vertical line” the indication is of “no benefit”.
You have to be a pretty ardent Neyman-Pearsonite to go along with that. The meta-analysis, of the available data, supports a small but beneficial effect – although it doesn’t make this “obvious” as Holford writes. The data do not support very big negative effects, or very big positive ones. The data are certainly also consistent with no effect – but that’s about as accurate as they allow you to be.
Now, if I was the FDA and terrified of making a Type I error in a final, binary decision, I’d accept the “no benefit” choice. But I’m not; and as nasty side-effects and/or massive cost of Omega 3 seem implausible, a more careful weighing of Type I/II errors seems appropriate.
More work also seems appropriate to figure out the possible confounding effect of publication bias in the smaller studies, or systematic bias in the big ones – for example why aren’t GISSI and Burr 2003 more concordant?
April 11, 2008 at 2:38 am
sorry, typo with “does cross the line”
April 11, 2008 at 9:21 am
Another discussion of the Hooper et al meta-analysis can be found in the paper by von Schacky and Harris here.
April 11, 2008 at 9:29 am
as an aside, I find it odd the Professor Holford of Teeside University did not contribute his insights to the BMJ’s rapid/rabid responses section, as he has been known to in the past (with predictable consequences.)
(Perhaps HW could remind readers of a time where Professor Holford has accused another scientist of vested interests, whilst regrettably forgetting to mention his own)
fascinating as his website may be, and as large as his mailing list no doubt is, Professor Holford must surely have a duty as a scientist (for he is very clear that he is, himself, a scientist) to raise these concerns in the published academic literature.
Rigorous, free intellectual discourse is the cornerstone of good science and Professor Holford should be encouraged to bring his observations of ‘obvious skew’ to the attention of his fellow scientists.
Perhaps Professor Holford has raised his allegations of “obvious skew” with Professor Summerbell, as their paths cross (literally, if not intellectually) on the campus of Teeside University?
April 11, 2008 at 11:07 am
Slightly off-topic, but: “…and as nasty side-effects and/or massive cost of Omega 3 seem implausible…”
Hm. Isn’t there at least a theoretical danger that there could be an increased risk of haemorrhagic stroke (or of bleeding to death after an accident)? That at least *seems* plausible to me, as n3 fatty acids reduce platelet aggregation – although Gerster seemed to think the concern over haemorrhagic stroke risk was unfounded.
I agree that the cost of fish oil should not be ‘massive’ – but could it be more cost-effective to get one’s Omega-3 from eating fish as part of a healthy balanced diet? The fish purchase would be part of your regular grocery bill, rather than involving an extra layout of cash.
Actually, you could eat 100g of tinned salmon per day for about £6 per month and that would provide ~1.1g EPA/DHA. The equivalent in fresh mackerel would be about £9 per month. The equivalent in capsules would cost £10 per month. Eating fish would seem to be a much more cost-effective way of obtaining Omega-3 than is consuming capsules. It’s funny how people are often prepared to spend more on pills than on food: http://www.badscience.net/?p=299
April 11, 2008 at 12:09 pm
I have a lecturer somewhere who groaning and saying, “I brought them up to be more hugger-mugger than proscriptive, where did it all go wrong?”
Are you OK with the later re-word of that point which is:
The authors of the review were surprised by their own results and commented on it in the Rapid Responses that follow.
The authors do comment on the various studies as the inclusion of Burr 2003 excited some controversy.
Overall my point is that there were no grounds for questioning the integrity of the authors and the journal.
April 11, 2008 at 12:11 pm
Ah, but you have to consider that the long term safety of mackerel has yet to be established. There’s evidence that mackerel consumption raises the risk of choking on a fish bone, and it’s been linked to your kitchen smelling a bit fishy all day.