Spinning Acupuncture

How should we read a scientific paper?

A new study, published last week¹, would suggest that it is a very bad idea to simply read the abstract and conclusions. Worse, it can be very misleading to rely on reports derived from press releases about the research. The study looked at how press releases included ‘spin’ about the results of a medical paper. The results were rather alarming.

The authors defined spin as the overestimation and overinterpretation of treatment benefits from RCTs. Unsurprisingly, they found press reports of medical trials contained significant spin. What was disturbing was that this spin was most closely correlated with spin found in the original papers’ abstracts.

We might expect newspapers to make medical research stories more sensational than they really are. Miracle cures and breakthroughs make better stories than mundane marginal improvements or negative results that are more likely in research. But what this story suggests is that we cannot reliably apply the tactic of looking for the original research as a corrective to media spin as the original papers are also likely to include the spin.

As most research is behind a paywall, most people only have access to abstracts of papers. A patient researching a treatment on the basis of a media report could well indeed remain misled if they can only access the abstracts of papers. Even, if we can access the full paper, it is a significant task to have to carefully read the full paper, appraise the presented data and come to our own conclusions regardless of what the abstract says.

In fact, this situation is all rather terrifying.

A good example of this came recently with a new systematic review² of studies looking at acupuncture for chronic pain. It concluded that “acupuncture is effective for the treatment of chronic pain and is therefore a reasonable referral option”.

The conclusion went on to say,

Acupuncture is effective for the treatment of chronic pain and is therefore a reasonable referral option. Significant differences between true and sham acupuncture indicate that acupuncture is more than a placebo. However, these differences are relatively modest, suggesting that factors in addition to the specific effects of needling are important contributors to the therapeutic effects of acupuncture.

And, as you might expect, acupuncturists and other supporters of superstitious and pseudoscientific treatments are proclaiming that they have been vindicated.

The study looked at 29 randomized clinical trials of acupuncture for various sorts of pain, including back, neck and headache. It looked at studies that used sham and no-acupuncture controls. Sham acupuncture is where the patient thinks they are getting acupuncture, but typically the needle is not being inserted, merely pressed against the skin. Vickers et al. reported that when acupuncture was compared with no- acupuncture, reported reduction in pain was fairly large, about 30%. However, crucially for supporters of acupuncture, they reported a moddest improvement of pain when ‘real’ acupuncture was compared with sham acupuncture.

And as such, the media reports of this finding (with a few honourable exceptions) have followed the abstracts conclusion that acupuncture is a valid treatment option for chronic pain. David Gorski at Science-based Medicine lists a number of these spun headlines.

Is it is as simple as that? Should we accept that indeed acupuncture can indeed by a useful treatment for pain. As you might expect, it is a lot more complicated than headlines would suggest.

First of all, if true, then this analysis shows something startling. Acupuncture claims to work through the needles unblocking something called qi that is supposed to flow through something called meridians. There are tens of thousands of specific points on the body that are supposed to respond to a fine needle being placed in them and wiggled around. This is straightforward pseudoscientific nonsense. Qi and meridians are vitalisitic notions without any evidence to support them. Modern acupuncture is largely a New Age invention that draws on much simpler Chinese ritualistic surgical techniques.

Indeed, one can think of acupuncture as a placebo-ised pre-scientific surgery. Just as homeopathy is a placebo-ised pre-scientific pharmacology, acupuncture is a stylised and minimalised form of blood letting, cauterisation, lancing and slicing. Indeed, the early texts that modern supporters of acupuncture claim to show it is thousands of years old describe a variety of blades and points that look indistinguishable from many societies primitive surgical kit bag.

So, for acupuncture to actually demonstrate specific effects would suggest that there is indeed some truth in what acupuncturists claim about how the body works and causes of illness.

There is good reason to believe that despite the study showing positive effects for real acupuncture over sham acupuncture that these are not due to genuine physiological effects. Blinding in trials of acupuncture is notoriously hard. Most people tend to know if a needle has been stuck in you. Sham stage-dagger-style acupuncture may be the most convincing placebo version of acupuncture we have. But the practitioners are not blinded as is usual in blinded experiments (hence ‘double-blind’). Any small leak of the blinding, either by the patient guessing whether the treatment is real or sham, or by clues given off by the acupuncturist, will lead to biases towards the real acupuncture appearing in the results. A small positive difference in the real acupuncture arm can be explained by small leaks in the blinding. The authors acknowledge this problem in their paper, but crucially, not in their abstract.

Similarly, while we considered the risk of bias of unblinding low in most studies comparing acupuncture and sham acupuncture, health care providers obviously were aware of the treatment provided, and, as such, a certain degree of bias of our effect estimate for specific effects cannot be entirely ruled out.

So, let’s step back. The acupuncture paper is indeed written by supporters of acupuncture. As Steven Novella writes,

I took a close look at the study and find that the authors display considerable pro-acupuncture bias in their analysis and discussion. They clearly want acupuncture to work. That aside, the data are simply not compelling, and the authors, in my opinion, grossly overcall the results, which are compatible with the conclusion that there are no specific effects to acupuncture beyond placebo.

Indeed, the journal the paper was written took the steps of publising an independent commentary³ on the paper, so surprising were the results. Avins notes correctly that the specific effects of acupuncture  are of ‘dubious clinical relevance’,

the authors state that the overall observed estimate of approximately 0.5 SD “is of clear clinical relevance,” but this assertion is difficult to substantiate.

Avins then attempts to move the debate on by arguing that despite the effects of acupuncture being placebo, that it may yet have relevance in clinical practice.

The conclusion, that most of acupuncture’s observed clinical benefit is mediated by placebo effects, will probably lead the factions to take their usual positions, with acupuncture advocates focusing on the usual-care comparisons and its detractors stressing the placebo nature of its effects. These latter critics correctly note that a new pharmaceutical agent that fails to show superiority over placebo will not be approved. Why should the bar be lowered for acupuncture? The authors acknowledge this discomfort, stating “ . . . many clinicians would feel uncomfortable in providing or referring patients to acupuncture if it were merely a potent placebo.” In response, they cite the small observed specific effect of acupuncture. But this line of argument is not persuasive: medications are developed and approved for their specific effects, and the small specific effects of acupuncture are of dubious clinical relevance. If we hold fast to the time-honored approach of placebo comparisons, then acupuncture’s detractors rightfully rule the day. But perhaps this debate should prompt us to take a careful look at the assumption that only specific biological effects matter and that the only valid estimate of effect is the comparison with placebo rather than with usual medical care.

This is a more nuanced position than was popularly reported in the press. The authors wanted the coverage to lead with the observed specific effects and that is the coverage they largely got. Acupuncturists are not happy with their techniques just being a placebo. They want more than that.

Nonetheless, despite this paper clearly showing the placebo nature of acupuncture is there still a role for it?

Much confusion surrounds discussions of placebo effects. Such effects may indeed lead to genuine subjective improvements in the experience of pain and so lead to a better quality of life – something to be welcomed. But placebo effects can also be little more than,measurement artifacts. For example, patients may be inclined to score their pain less in order to reciprocate good will towards their acupuncturist. The expectations of improvement and natural desire to please may introduce biases in score results towards positive effects. This does not mean that the patient will continue to receive benefits over any meaningful period of time, if they received any at all. Care is required in understanding just what has occured when a placebo effect is reported.

And then there is the difficult problem of offering placebo treatments in general. Either the practitioner has to intentionally deceive the patient and so undermine some pretty fundamental issues of trust, autonomy and informed consent in medicine. Or the acupuncturist has to genuinely believe in the specific effects of acupuncture and so, in a real sense, be systematically incompetent in their practice.

The problem of True Believers offering a placebo treatment such as acupuncture has other risks. Acupuncturists believe that their needles can indeed treat all sorts of conditions from infertility, smoking cessation, tinnitus to depression. Pain is a placebo responsive condition. Leading patients to believe that acupuncture has specific effects for pain, may reduced their ability to make informed decisions about other health care issues.

And then there are the risks of acupuncture. Any improvements gained from acupuncture, placebo or not, need to be weighed against the risks. A recent paper⁴ shows that NHS acupuncture indeed carries significant risks from pneumothorax to needles being left in the body. Although rare, the authors worry that adverse events from acupuncture are under-reported. There is an intrinsic belief in acupuncture (like a lot of alternative medicine) that the treatments are inherently safe and natural. The need for systematic reporting of problems is not seen as a priority.

Even with this review of acupuncture that failed to show convincing specific effects for chronic pain, we are likely to see a large increase of, not just patients, but also healthcare providers that now view acupuncture as a genuine referral option. The reasons for this are not because of convincing and thorough data, but because of specific spin placed on the study by its authors, dutifully copied by journalists around the world.

And whose fault is this? The Yavchitz paper on misprepresentaion of RCTs is clear: journal reviewers and editors,

Overall, these findings highlight the important role that journal reviewers and editors play in disseminating research findings. These individuals, the researchers conclude, have a responsibility to ensure that the conclusions reported in the abstracts of peer-reviewed articles are appropriate and do not over-interpret the results of clinical research.

¹ Yavchitz A, Boutron I, Bafeta A, Marroun I, Charles P, et al. (2012) Misrepresentation of Randomized Controlled Trials in Press Releases and News Coverage: A Cohort Study. PLoS Med 9(9): e1001308. doi:10.1371/journal.pmed.1001308

² Vickers AJ, Cronin AM, Maschino AC, et al. Acupuncture for Chronic Pain: Individual Patient Data Meta-analysis. Arch Intern Med. Published online September 10, 2012. doi:10.1001/archinternmed.2012.3654.

³ Avins AL. Needling the Status Quo: Comment on “Acupuncture for Chronic Pain”. Arch Intern Med. Published online September 10, 2012. doi:10.1001/archinternmed.2012.4198.

⁴ Risks of acupuncture range from stray needles to pneumothorax, finds study
BMJ 2012; 345 doi: 10.1136/bmj.e6060 (Published 7 September 2012)

Follow Up

6th March 2013

Incredibly, the authors of this paper have written another paper moaning about the response of bloggers and skeptics.

The abstract reads,

In September 2012 the Acupuncture Trialists’ Collaboration published the results of an individual patient data meta-analysis of almost 18 000 patients in high quality randomised trials. The results favoured acupuncture. Although there was little argument about the findings in the scientific press, a controversy played out in blog posts and the lay press. This controversy was characterised by ad hominem remarks, anonymous criticism, phony expertise and the use of opinion to contradict data, predominantly by self-proclaimed sceptics. There was a near complete absence of substantive scientific critique. The lack of any reasoned debate about the main findings of the Acupuncture Trialists’ Collaboration paper underlines the fact that mainstream science has moved on from the intellectual sterility and ad hominem attacks that characterise the sceptics’ movement.