What many people don’t realize about scientific studies is that since they are designed, conducted, analyzed, and reported by fallible human beings, they are prone to all sorts of bias and error. They often contain mechanisms to minimize these sources of error, which is why they are still more reliable than personal experience, history, and other uncontrolled sources of information. And, of course, the best compensation for the failings of individual scientists is the work of other scientists, critiquing the work, trying to replicate it, and generally bashing it about until the truth falls out. Science is a community endeavor, and the community keeps the individual honest. At least, that’s the theory.
However, some communities prize this sort of critical, competitive error correction process more than others. I have written often about the general tendency of people in the alternative medicine community to prefer unity and validation of each other’s theories to rigorous, skeptical scrutiny aimed at paring away bias and error. As a category, Alternative Medicine only exists to protect some practices from the standards of evidence scientific medicine are expected to conform to. If these therapies can prove themselves by accepted scientific means, they are not “alternative” or “complementary” and wouldn’t need to be “integrated” with regular medicine because they would simply be regular medicine.
One example of the misuse of science to confirm and support rather than challenge excepted beliefs is the literature concerning acupuncture. The vast majority of acupuncture studies are done in countries where it is a widely accepted practice (though not as widely as sometimes claimed), and where most practitioners and others already accept its effectiveness. China, in particular, contributes a tremendous percentage of the research on acupuncture. And as I’ve discussed before, there is strong evidence that Chinese acupuncture studies are biased in favor of acupuncture. This evidence includes studies which have shown negative results of acupuncture research are almost never published, studies are often inaccurately reported as randomized when they aren’t, and systematic reviews often selectively search and report the literature in ways that are favorable to acupuncture. Yet another study has now been published which confirms that Chinese researchers simply do not produce or report negative results for acupuncture.
Yuyi Wang, Liqiong Wang, Qianyun Chai, Jianping Liu. Positive results in randomized controlled trials on acupuncture published in chinese journals: a systematic literature review. J Altern Complement Med 2014 May;20(5):A129
This review found 847 reported randomized clinical trials of acupuncture in Chinese journals. 99.8% of these reported positive results. Of those that compared acupuncture to conventional therapies, 88.3% found acupuncture superior, and 11.7% found it as good as conventional treatments. Very few of the studies properly reported important markers of quality and control for bias such as blinding, allocation concealment, and losses to follow-up.
Of course, one could argue that the failure to publish negative results, and the overwhelming superiority of acupuncture compared with conventional treatments is evidence that acupuncture is incredibly effective and that Chinese researchers do a nearly perfect job of employing it. That seems a pretty implausible interpretation, however, It would suggest that acupuncture is unlike any other therapy ever tested scientifically, and that Chinese acupuncturists are nearly perfect clinicians. It would also beg the question of why acupuncture was practiced in one form or another for thousands of years without meaningfully improving the life-expectancy or mortality patterns of people in China while science-based medicine has dramatically extended life and reduced disease there as everywhere else.
A more likely interpretation of this and the other studies showing that the Chinese almost never report failures in acupuncture treatment is simply that the design, conduct, and reporting of these studies is biased towards supporting the already widespread belief that acupuncture works. Belief trumps and distorts science all the time, and this is likely yet another example of this. All kinds of cultural theories can be advanced to explain these findings, and the differences between the acupuncture literature in China and that in the English language literature, where negative studies are much more common. I am no sociologist, but I do know that science exists specifically as a method for combating the natural human tendency to seek confirmation rather than refutation of our existing beliefs, and that no system for checking human bias can be successful without an explicit commitment to following the methods and accepting the results even when they are not consistent with what we want to believe.
I have talked previously about the dangers of alternative medicine research functioning as marketing and propaganda rather than a careful and genuine effort to seek the truth. The analysis of the Chinese literature pertaining to acupuncture, and most of the literature related to homeopathy, illustrate this danger. It is imperative that scientific evaluation of alternative therapies be held to at least as high a standard as research on conventional treatments is in order to prevent people, and our pets, from being subjected to ineffective or unsafe therapies under the misguided belief that they have been proven to work.
