BSAVA Statement on Complementary & Alternative Medicine

Organizations representing veterinarians are fundamentally political in nature, and their leaders respond to the will of their constituencies. I have often written about the American Veterinary Medical Association (AVMA) in the context of alternative therapies and evidence-based medicine. While the organization has adopted some policies I believe are scientifically sound and in the best interests of veterinary patients and clients as well as veterinarians, there is no question that the group answers to veterinarians first and foremost, even when these conflict with the scientific facts or the interests of clients and patients. And within the veterinary profession, there are different and often competing constituencies that leaders of groups like the AVMA must respond to.

In regards to complementary and alternative veterinary medicine, CAVM, I have often been disappointed in the AVMA’s positions, though I understand the political realities that shape them. There is a constituency of pro-CAVM veterinarians which is small but vocal and well-funded. The constituency devoted to science and evidence-based medicine is smaller, less vocal, and not nearly as well-funded. And the large majority of veterinarians appear to have little interest or opinion concerning CAVM. This leads the AVMA as a whole to rather tepid positions on CAVM that reflect the political landscape far more than the scientific evidence.

The handling of the resolution introduced into the AVMA House of Delegates in 2012 discouraging the use of ineffective therapies, and specifically identifying homeopathy as such a therapy, illustrates the politics of CAVM within the AVMA. The science conclusively shows homeopathy to be nothing more than a placebo, despite attempts by homeopaths to claim otherwise, and the AVMA’s Council on Research supported this conclusion. However, the House of Delegates voted overwhelmingly against the resolution anyway because of vociferous opposition from pro-CAVM veterinarians, a general lack of interest in the scientific issues on the part of most members, and a deep cultural reluctance among veterinarians to criticize the practices of colleagues.

The AVMA also has a general statement on CAVM which used to be a fairly thoughtful, though still politically circumspect, document. It at least identified some core issues raised by the use of CAVM:

The theoretical bases and techniques of CAVM may diverge from veterinary medicine routinely taught in North American veterinary medical schools or may differ from current scientific knowledge, or both.

The AVMA believes that all veterinary medicine, including CAVM, should be held to the same standards. Claims for safety and effectiveness ultimately should be proven by the scientific method….Practices and philosophies that are ineffective or unsafe should be discarded.

The AVMA does not officially recognize diplomate-status or certificates other than those awarded by veterinary specialty organizations that are members of the AVMA American Board of Veterinary Specialties (ABVS), nor has it evaluated the training or education programs of other entities that provide such certificates. Recognition of a veterinary specialty organization by the AVMA requires demonstration of a substantial body of scientific knowledge. The AVMA encourages CAVM organizations to demonstrate such a body of knowledge.

The quality of studies and reports pertaining to CAVM varies; therefore, it is incumbent on a veterinarian to critically evaluate the literature and other sources of information. Veterinarians and organizations providing or promoting CAVM are encouraged to join with the AVMA in advocating sound research necessary to establish proof of safety and efficacy.

In this document, the AVMA refrained from passing judgment on the scientific merits of CAVM, but it at least acknowledged that science was the standard by which veterinary therapies should be judged and that, as of yet, few CAVM practices have met such a standard.

The new AVMA position on CAVM has been stripped of essentially all content and says merely that all therapies should be held to the same standard (without indicating what that standard should be) and that nobody should break the law. This revised statement moves further away from the notion that veterinary medicine is, or should be, a science-based profession. It reflects the growing political reluctance to make critical, evidence-based judgments about claims or practices so long as a licensed veterinarian is employing them.

Other veterinary organizations have been more faithful to the role of science as the foundation of veterinary medicine, with many specifically acknowledging the unacceptability of homeopathy in a science-based profession, and many identifying science and evidence-based medicine as essential to veterinary medicine.

A recent statement on CAVM from the British Small Animal Veterinary Association (BSAVA) takes a fairly moderate, political position on the issue of CAVM, though still one more robust than that offered by the AVMA. Like the previous AVMA statement, the BSAVA document acknowledges science and scientific research as the essential foundation for veterinary medicine, and explicitly endorses and evidence-based approach:

The BSAVA recommends that owners consider the evidence for a particular treatment and the qualifications and experience of the practitioner before embarking on any complementary or alternative therapy for their pet.

The BSAVA strongly recommends that whenever possible treatment decisions are based on sound scientific evidence to support the safety and efficacy of the therapy.

Assessing the evidence

Some people, including owners, therapists and veterinary surgeons, may perceive that the therapies work as a result of belief in the therapy (placebo effect); anecdotal evidence (extrapolation from hearsay or personal experience of a single or small number of cases) or errors in inference (cognitive bias).

There are three factors strongly associated with whether or not any one medical treatment is likely to be efficacious:

1. A rational scientific basis

Modern medicine works, and it works because it is founded on a scientific base. Although not all treatments used in conventional medicine have a strong evidence base in the sense of rigorous clinical trials showing their efficacy they do have a rational scientific / pathophysiological basis for their use.

2. Degree of certainty

The effects of some treatments are so clear cut that further testing is not required. It has famously been pointed out that rigorous clinical trials are not needed to prove that parachutes reduce morbidity and mortality among people falling from aeroplanes. Similarly, one does not need rigorous trials to show that intravenous anaesthetics cause a rapid, profound loss of consciousness suitable for carrying out surgery. However, the effects of many treatments are much less certain; e.g., they are less closely associated in time with their effect, or the effect caused is much less dramatic, smaller and/or more variable in magnitude and/or time of onset. In such circumstances, given the variable time courses of many diseases, it can be remarkably difficult to determine whether a given treatment is actually efficacious or not.

3. Evidence

When there is anything less than absolute certainty about the efficacy of a treatment, then evidence is important in deciding whether a treatment is safe and efficacious. However, history has also shown that evidence – both in the form of clinical experience and individual clinical research results – can be misleading. The process of evidence-based (veterinary) medicine exists to improve our confidence by formally and systematically searching for all of the relevant evidence and formally and systematically grading the quality and reliability of that evidence.

The organization also acknowledges that the evidence for most CAVM is poor and that safety or efficacy is largely unknown for many CAVM practices:

Health claims for many complementary and alternative therapies are far in excess of the available scientific data, and sometimes in frank contradiction to scientific evidence.

In making decisions about the use of complementary and alternative therapies it is important to consider their safety and efficacy. Many people assume that all complementary and alternative therapies are natural and therefore safe, but this is not always the case. All therapies may produce unwanted side effects or may interact with other therapies. In the case of alternative therapies it is also important to consider the welfare implications of withholding conventional treatments.

There is a great deal of variation in both the degree to which various complementary and alternative therapies have been scientifically tested, and to which such testing has provided evidence supporting their efficacy.

While I believe it would have been appropriate to go further and acknowledge that at least some CAVM practices are completely without merit and should be discouraged, such as homeopathy and so-called “energy” therapies, I think this is a solid statement supporting the important role of science in veterinary medicine. Relying on science to understand and predict nature has led to far greater health and well-being and far less physical suffering than human beings, and our animal companions, experienced in all the thousands of years before we developed the scientific method. Acknowledging the unparalleled power of science to guide us to the right answers in healthcare is essential in maintaining and extending the gains science has allowed medicine to achieve.

This statement seems to indicate that the constituency of the BSAVA, like that of other veterinary groups in Europe and Australia, is moving in the right direction with respect to deepening the reliance on science and evidence-based medicine in the veterinary field. Sadly, the recent actions of the AVMA would suggest that here in America, as seems evident in many other areas, we are moving in the wrong direction, away from rationalism and science and towards an epistemology that privileges personal experience and belief over objective, scientific evidence. This is not in the best interests of our patients, our clients, or our profession.

Posted in Law, Regulation, and Politics | 1 Comment

JustFoodForDogs Brings Us Some Classic Marketing Masquerading as Science

I have written about the issue of homemade pet diets here several times (1, 2, 3, 4). They are appealing to some owners because they appear more “natural” than commercial dry or canned diets, which is supposed to imply they are better for pets. And, of course, many proponents of alternative medicine make hysterical and unsupported accusations about the dangers of conventional pet diets.

People also equate conventional commercial pet food with what is typically called “processed food,” though they are entirely different things. Human snack foods and other processed foods are laden with excessive sugar, salt and fat and generally nutritionally poor. Commercial pet foods, if properly formulated and manufactured, are nutritionally balanced to a greater degree than our haphazard diet of whatever looks appealing in the moment, even when we the packaged junk foods are avoided.

Homemade diets can be perfectly healthy, and there are circumstances in which a diet formulated for the specific needs of a particular pet is better than any commercially available diet. And fresh food is certainly attractive to many pets. But the dramatic claims of health benefits made for them are entirely unproven, and the existing research suggests most recipes for homemade diets, even those promoted by veterinarians, are not appropriately balanced nutritionally and not ideal for long-term health. I encourage anyone interested in preparing food at home for their pets to consult with a board-certified veterinary nutritionist for guidance.

I recently ran across a press release from a company which appears to be trying to cash in on fears of commercial pet food and the appeal of homemade diets in order to sell—you guessed it, their commercial pet food. Reminiscent of the “just like homemade” marketing approach often used to sell packaged foods for people,  JustFoodForDogs makes heavy use of terms like “scientific” and “evidence” in their marketing to suggest that dry commercial diets and the ingredients they contain are unsafe and that their packaged frozen cooked diets are better. While these diets appear to meet all the same standards for balanced nutrition of other commercial diets, including AAFCO feeding trial tests, the evidence offered for their superiority is so far scant.

One unpublished study funded by the company and run by one of their veterinarians is referred to in their press release as “groundbreaking” and “game changing.” Science by press release is always a bad sign (anybody remember “cold fusion?”), but the presentation of the study is clearly designed to maximize its marketing value without providing any of the information that would be needed to determine if the methods were really appropriate.

Twenty-one dogs of unspecified breeds were fed some of the company’s diets (the details are not reported) and basic bloodwork and exams were conducted at the beginning of feeding the diet and again at 6 months and twelve months. No control group, no blinding, no pre-specified outcomes or hypotheses, no reported accounting for repeated measures or multiple comparisons in the statistical analysis, no discussion of any other aspects of the dogs health or environment, and overall none of the hallmarks of an actual controlled clinical study. All of this would be fine of the purpose were merely to explore the effects of the diet and generate hypotheses. But the company clearly intends to present these results as earthshattering, paradigm-shifting research that (coincidentally?) favors their product.

And after setting up everything with no apparent effort to control for the obvious risk of bias, what were the reported results? One kind of blood protein, globulins, went up (by how much isn’t disclosed). Some kinds of white blood cell numbers increased (again, by how much isn’t disclosed, but the numbers were apparently still within the range of normal). And one measure of red blood cells increased, though another did not.

Given the comparisons of many different values with no explicit reason and no reported use of statistical methods to control for making them, it is almost guaranteed some values would change to a degree judged “statistically significant.” This is not the same thing as medically significant, and there is no evidence these changes had any clinical relevance, especially with no control group for comparison. But the company promotes the results as showing their foods “could benefit immune health” and that if the purported trends in the blood values continue for the animals’ lifetimes “we may see a decrease in chronic diseases such as cancer, renal failure, kidney disease, inflammatory bowel disease, dental disease, etc.”These results certainly don’t support anything even approaching such claims.

The hypothesis that fresh foods could have health advantages over extruded kibble or commercial diets is not an unreasonable one, and I am open to the possibility this might be true. But this is not something we can simply assume without evidence, and that evidence does not yet exist. Furthermore, the claims made about the dangers of conventional commercial diets are rarely supported by evidence either, whereas there is abundant scientific research and real-world experience showing that pets can live long, health lives on these foods.

If the folks behind this company genuinely believe their claims about health and nutrition, and I have no reason to think they don’t, then they should make an effort to design and conduct properly controlled scientific research to evaluate their hypotheses. But they do a disservice to pets and pet owners when they perform “studies” clearly designed with marketing rather than science in mind, hype the results to an extreme degree, and then use this as a marketing strategy to promote their own products.

Posted in Nutrition | 82 Comments

Liberals not Immune to Science Denialism

This should be mandatory viewing in every Starbucks, Pilates class, and Whole Foods in America!

 

 

Posted in Humor | 1 Comment

American Holistic Veterinary Medical Association (AHVMA) & its Foundation (AHVMF)

I recently received a registration packet for the 2014 Annual Conference of the American Holistic Veterinary Medical Association. This seemed a good reminder to pull together some of the information and observations concerning organized alternative veterinary medicine I have posted in the past.

I have written frequently about the American Holistic Veterinary Medical Association (AHVMA) and the spinoff group the American Holistic Veterinary Medical Foundation (AHVMF). These organizations are the tip of the spear, so to speak, of the effort to promote alternative therapies in veterinary medicine. They mimic the organizational structure and functions of the American Veterinary Medical Association (AVMA) and many other veterinary professional organizations, and they appear on the surface no different. In fact, the AHVMA recently became large enough to qualify for a seat in the governing House of Delegates of the AVMA as an affiliate organization.

These groups are very successful financially (AHVMA and AHVMF financial statements), with combined revenue in 2012 of over a million dollars. These resources give them influence, as illustrated by their donations to veterinary medical colleges intended to create or support the integration of alternative therapies into the curriculum. In 2012, the AHMVF announced a grant of $10,000 to the University of Tennessee to support an integrative medicine program, and another $40,000 grant to support an integrative medicine fellowship (though the tax filing only lists the $40,000 amount). The same year, the AHVMF announced a $200,000 grant to the University of Louisiana, for the formation of an integrative medicine program and the hiring of a faculty member trained in Chinese Veterinary Medicine (though again, the tax filing reported different numbers, with only $110,000 given to the university). They also reported over $13,000 in scholarships for students to support studying alternative veterinary medicine.

All of this is a good thing, if you believe that alternative/holistic/integrative medicine is a collection of safe and effective approaches to healing pets. If, however, you are committed to science as the best way to understand nature and to develop safe and effective medicine, well the success of these organizations should be troubling. The activities of these groups and the statements of their leaders have consistently demonstrated a superficial respect for scientific methods masking a deep philosophical rejection of the basic principles of science and evidence-based medicine. Yet their activities, and the influence of their financial resources, create the appearance of legitimacy for many therapies that are at least dubious and unproven, and often complete quackery.

Both organizations share a clear mission of advocacy and promotion of alternative therapies. While the AHVMF often claims to be interested in research into alternative approaches, it is clear that approach here is to use science the way a drunk uses a lamppost–for support, not illumination. The goal is to generate the appearance of positive scientific research findings and validation, but there is no serious willingness to follow where the evidence leads or reject practices when the scientific evidence against them is overwhelming. This is made starkly clear by the aggressive lobbying of the AHVMA against the resolution considered last year by the AVMA to declare homeopathy an ineffective therapy. Despite the clear scientific consensus that after 150 years of study homeopathy has been proven to be nothing more than a placebo, the AHVMA stood behind a clearly misleading collection of pseudoscience put forward by the Academy of Veterinary Homeopathy (AVH), and proudly trumpeted the defeat of the resolution as a victory.

I have reviewed the proceedings available for a couple of AHVMA meetings in the past, 2009 and 2012. While there were sessions on some promising but as yet unproven methods (for example herbal remedies and dietary supplements and cold laser therapy), there were many on the Big Three of alternative medicine (acupuncture, chiropractic, and homeopathy), and plenty on other varieties of nonsense, from Traditional Chinese Veterinary Medicine to shamanism, to Earth Acupuncture, and so on. The tone of the majority of speakers clearly showed a deep philosophical rejection of science in favor of vitalism and the tyranny of the anecdote and personal experience. The offerings at the 2014 conference appear to be much the same, with a few new ”hot topics” (leech therapy, bee venom therapy, and medical uses of marijuana, for example).

I have also previously written about the funding for these meetings, which provides a major source of income for the AHVMA. They draw sponsorship from a wide range of commercial organizations, mostly those providing products and services used by alternative veterinary practitioners. These include manufacturers and distributers of herbal products, dietary supplements, laser equipment, unconventional foods such as raw diets, and companies which teach alternative therapies, such as the Chi Institute (the full list for 2104 is available here).

The point in mentioning this is not to suggest that these groups should do without such funding sources. The reality is that no veterinary education or outreach activity can occur without some source of funding, and that inevitably means working with related industries. However, the potential influence of this money on the practices employed by veterinarians and on the generation and interpretation of scientific evidence concerning these is a legitimate concern. Proponents of alternative medicine are quick to remind us of this when criticizing the use of pharmaceuticals and the potential influence of Big Pharma on conventional doctors, but they seem reluctant to acknowledge that their own activities are no freer from the influence of commercial funding.

Because the AHVMA and AHVMF are influential organizations and, in my opinion, are primarily promoting pseudoscience and anti-science in veterinary medicine, I try to keep track of their activities. So I have added this post to my list of topic-based summaries and will try to keep an up-to-date collection of relevant posts here.

Posts Related to the AHVMA and AHVMF

Dr. Nancy Scanlan Shows us How to Talk Sciency Without Really Accepting Science

Dr. Barbara Royal Reminds us that the AHVMF Opposes Science-Based Medicine

The 2012 AHVMA Annual Conference: An illustration of Conflicts between Science-Based Medicine and Holistic Veterinary Medicine

Leader of Holistic Veterinary Foundation Express some Troubling Ideas about Science

American Holistic Veterinary Medical Foundation gives $10,000 to University of Tennessee Veterinary School to Promote Alternative Medicine

Response to Comments from the American Holistic Veterinary medical Association on the AVMA Homeopathy Resolution

The American Holistic Veterinary Medical Foundation (AHVMF): Science of Salesmanship?

Politics Trumps Science: Continuing Education Credit for Pseudoscience Thanks to the AHVMA

The AHVMA: Bought and Paid for by Big Supplement?

“Holistic Medicine:” It Means Whatever We Say It Means

Woo U. — CAVM as Continuing Education for Veterinarians

 

Posted in Topic-Based Summaries | 13 Comments

Evidence Update-Chinese Studies of Acupuncture Are Always Positive: Perfect Medicine or Hidden Bias?

What many people don’t realize about scientific studies is that since they are designed, conducted, analyzed, and reported by fallible human beings, they are prone to all sorts of bias and error. They often contain mechanisms to minimize these sources of error, which is why they are still more reliable than personal experience, history, and other uncontrolled sources of information. And, of course, the best compensation for the failings of individual scientists is the work of other scientists, critiquing the work, trying to replicate it, and generally bashing it about until the truth falls out. Science is a community endeavor, and the community keeps the individual honest. At least, that’s the theory.

However, some communities prize this sort of critical, competitive error correction process more than others. I have written often about the general tendency of people in the alternative medicine community to prefer unity and validation of each other’s theories to rigorous, skeptical scrutiny aimed at paring away bias and error. As a category, Alternative Medicine only exists to protect some practices from the standards of evidence scientific medicine are expected to conform to. If these therapies can prove themselves by accepted scientific means, they are not “alternative” or “complementary” and wouldn’t need to be “integrated” with regular medicine because they would simply be regular medicine.

One example of the misuse of science to confirm and support rather than challenge excepted beliefs is the literature concerning acupuncture. The vast majority of acupuncture studies are done in countries where it is a widely accepted practice (though not as widely as sometimes claimed), and where most practitioners and others already accept its effectiveness. China, in particular, contributes a tremendous percentage of the research on acupuncture. And as I’ve discussed before, there is strong evidence that Chinese acupuncture studies are biased in favor of acupuncture. This evidence includes studies which have shown negative results of acupuncture research are almost never published, studies are often inaccurately reported as randomized when they aren’t, and systematic reviews often selectively search and report the literature in ways that are favorable to acupuncture. Yet another study has now been published which confirms that Chinese researchers simply do not produce or report negative results for acupuncture.

Yuyi Wang, Liqiong Wang, Qianyun Chai, Jianping Liu. Positive results in randomized controlled trials on acupuncture published in chinese journals: a systematic literature review. J Altern Complement Med 2014 May;20(5):A129

This review found 847 reported randomized clinical trials of acupuncture in Chinese journals. 99.8% of these reported positive results. Of those that compared acupuncture to conventional therapies, 88.3% found acupuncture superior, and 11.7% found it as good as conventional treatments. Very few of the studies properly reported important markers of quality and control for bias such as blinding, allocation concealment, and losses to follow-up.

Of course, one could argue that the failure to publish negative results, and the overwhelming superiority of acupuncture compared with conventional treatments is evidence that acupuncture is incredibly effective and that Chinese researchers do a nearly perfect job of employing it. That seems a pretty implausible interpretation, however, It would suggest that acupuncture is unlike any other therapy ever tested scientifically, and that Chinese acupuncturists are nearly perfect clinicians. It would also beg the question of why acupuncture was practiced in one form or another for thousands of years without meaningfully improving the life-expectancy or mortality patterns of people in China while science-based medicine has dramatically extended life and reduced disease there as everywhere else.

A more likely interpretation of this and the other studies showing that the Chinese almost never report failures in acupuncture treatment is simply that the design, conduct, and reporting of these studies is biased towards supporting the already widespread belief that acupuncture works. Belief trumps and distorts science all the time, and this is likely yet another example of this. All kinds of cultural theories can be advanced to explain these findings, and the differences between the acupuncture literature in China and that in the English language literature, where negative studies are much more common. I am no sociologist, but I do know that science exists specifically as a method for combating the natural human tendency to seek confirmation rather than refutation of our existing beliefs, and that no system for checking human bias can be successful without an explicit commitment to following the methods and accepting the results even when they are not consistent with what we want to believe.

I have talked previously about the dangers of alternative medicine research functioning as marketing and propaganda rather than a careful and genuine effort to seek the truth. The analysis of the Chinese literature pertaining to acupuncture, and most of the literature related to homeopathy, illustrate this danger. It is imperative that scientific evaluation of alternative therapies be held to at least as high a standard as research on conventional treatments is in order to prevent people, and our pets, from being subjected to ineffective or unsafe therapies under the misguided belief that they have been proven to work.

Posted in Acupuncture | Leave a comment

Update-Do Dogs Defecate in Alignment with the Earth’s Magnetic Field?

Earlier this year, I reviewed a research study claiming that dogs orient themselves to fluctuations in the earth’s magnetic field when defecating. I was asked to re-post the review on Publons, a site which publishes reviews of journal articles. The authors have posted a response there which answers some concerns I expressed in my review, but which also illustrates some of the same misconceptions about statistics and hypothesis testing that I originally discussed. I have responded both here and on the Publons forum.

The summary of the reactions of the media on our paper is very fitting and we agree. The critic of our study is, however, biased and indicates that the author did not read the paper carefully, misinterpreted it in some cases, and, in any case is so “blinded” by statistics that he forgets biology. Statistics is just a helpful mean to prove or disprove observed phenomena. The problem is that statistics can “prove” phenomena and relations which actually do not exist, but it can also “disprove” phenomena which objectively exist. So, not only approaches which ignore proper statistics might be wrong but also uncritical sticking on statistical purity and ignoring real life.

To begin with, I believe the author and I agree that statistics are easily and commonly misused in science. Unfortunately, this response seems to perpetuate some of the misconceptions about the role of statistics in testing hypotheses I discussed in my original critique.

Statistics never prove or disprove anything. Schema such as Hill’s Criteria of Causation and other mechanisms for evaluating the evidence for relationships observed in research studies illustrate the fact that establishing the reality of hypothesized phenomena in nature is a complex business that must rest on a comprehensive evaluation of many different kinds of evidence. It is unfortunate that p-values have become the sine qua non of validating explanations of natural phenomena, at least in medicine (which is the domain I am most familiar with). The work of John Ionnidis and the growing interest in Bayesian statistical methods are examples of the move in medical research to address the problem of improper use and reliance on frequentist statistical methods.

That said, these methods do have an important role in data analysis, and they contribute significantly to our ability to control for chance and other sources of error in research. The proper role of statistical hypothesis testing is to help assess the likelihood that our findings might be due to chance or confounding variables, which humans are notoriously terrible at recognizing. If we employ these tools improperly, then they cease to fulfill this function and instead they generate a false impression of truth or reliability for results that may easily be artifacts of chance or bias.

The authors accuse me of being “so ‘blinded’ by statistics that he forgets biology.” This is ironic since their paper uses statistics to “prove” something which a broader consideration of biology, evolution, and other information would suggest is improbable. Even if the statistical methods were perfectly and properly applied, they would not be “proof” of anything any more than improper use of statistics would be definitive “disproof” or the authors’ hypothesis. While I discussed some concerns about how statistics were used in the paper, my objections were broader than that, which the authors do not appear to acknowledge.

The author of this critic blames us of “data mining”. Well, first we should realize that there is nothing wrong about data mining. This is an approach normally used in current biology and a source of many interesting and important findings. We would like to point out that we have not “played” with statistics in order to find out eventually some “positive” results. And we have definitively not sorted data out. We just tested several hypotheses and always when we rejected one, we returned all the cards (i.e. data) into the game and tested, independently, anew, another hypothesis.

Though I am not a statistician, I believe there is a consensus that while exploratory analysis of data is, of course, appropriate and necessary, the post-hoc application of statistical significance tests to data after patterns in the data have already been observed is incorrect and misleading. This is what the paper appeared to suggest was done, and this would fit the definition of inappropriate data-dredging.

Note also that we performed this search for the best explanation in a single data sample of one dog only, the borzoi Diadem, for which we had most data. When we had found a clue, we tested this final hypothesis in other dogs, now without Diadem.

This was not indicated in the description of the methods provided in the original paper. If the exploratory analysis was done with one data set while the authors remained blind to the data set actually analyzed in the paper, then that would be an appropriate method of data analysis. The subsequent statistically significant results would not, of course, necessarily prove the hypothesis to be true, but they would at least reliably indicate the likelihood that they were due solely to chance effects.

This does not, however, entirely answer the concern that the study began without a defined hypothesis and examined a broad range of behaviors and magnetic variables in order to identify a pattern or relationship. As exploratory, descriptive work this is, of course, completely appropriate. But the authors then use statistical hypothesis testing to support very strong claims to have “proven” a hypothesis not even identified until after the data collection was completed. This seems a questionable way to employ frequentist statistical methods.

Let us illustrate our above arguments about statistics and “real life” on two examples. Most medical diagnoses are done through exclusion or verification of different hypotheses in subsequent steps. Does it mean that when the physician eventually finds that a patient suffers under certain illness, the diagnosis must be considered improbable because the physician has already before tested (and rejected) several other hypotheses?

This analogy is inapplicable. The process of inductive reasoning a clinician engages in to seek a diagnosis in an individual patient is not truly analogous to the process of collecting data and then evaluating it statistically to assess the likelihood that patterns seen in the data are due to chance. Making multiple statistical comparisons, particularly after one has already sought for patterns in the data, invalidates the application of statistical hypothesis testing. The fact that in other contexts, and without the use of such statistical methods, people consider possible explanations and then accept or reject them based on their observations is irrelevant.

Or imagine that we want to test the hypothesis that the healthy human can run one kilometer with an average speed of 3 m/s. We find volunteers all over the country who should organize races and measure the speed. We shall get a huge sample of data, we have an impression that our hypothesis is correct but the large scatter makes the result insignificant. So we try to find out what could be the factors influencing speed. We test the age – and find out that indeed older people are slower than younger ones, so we divide the sample into age categories, but the scatter is still too high, so we test the effect of sex, we find a slight influence, but it still cannot explain the scatter, we test the position of the sun and time of the day, but find no effect, we test the effect of wind, but the wind was weak or it was windless during races, so we find no effect. We are desperate and we visit the places where the races took place – and we find the clue: some races were done downhill (and people ran much faster), some uphill (and people ran much slower), those who ran in flat land ran on average with the speed we expected. So we can now conclude that our hypothesis was correct and moreover we found an effect of the slope on running speed. We publish a paper describing these findings and then you publish a critic arguing that our approach was just data mining and was wrong and hence our observation is worthless and that the slope has no effect on running speed at all. Absurd!

Again, this example simply describes a process for considering and evaluating multiple variables in order to explain an observed outcome, which is not the objection raised to the original paper. If the only hypothesis in a study such as described here was that at least one human being could run this fast, then a single data point would be sufficient proof and statistics would be unnecessary. However, if one is trying to explain differences in the average speed of different groups of people based on the sorts of variables mentioned, the reliability of the conclusions and the appropriateness of the statistical methods used would depend on how the data was collected and analyzed. In any case, nothing about this has any direct relevance to whether or not the data collection and analysis in the original paper was appropriate or justified the authors’ conclusions.

As I said in the original critique, this study raises an interesting possibility; that dogs may adjust their behavior to features of the magnetic field of the earth. The study was clearly a broadly targeted exploration of behavior and various features of the magnetic environment: “we monitored spontaneous alignment in dogs during diverse activities (resting, feeding and excreting) and eventually focused on excreting (defecation and urination incl. marking) as this activity appeared to be most promising with regard to obtaining large sets of data independent of time and space, and at the same time it seems to be least prone to be affected by the surroundings.” It did not apparently start with a specific, clearly defined hypothesis and prediction, so in this sense it seems an interesting exploratory project.

However, with such a broad focus, with mostly post-hoc hypothesis generation, and with a lack of clear controls for a number of possible alternative explanations, the study cannot be viewed as definitive “proof” of the validity of the explanation the authors provide for their observations, though this is what is claimed in the paper: “…for the first time that (a) magnetic sensitivity was proved in dogs, (b) a measurable, predictable behavioral reaction upon natural MF fluctuations could be unambiguously proven in a mammal, and (c) high sensitivity to small changes in polarity, rather than in intensity, of MF was identified as biologically meaningful.”

I agree with the authors that their results are interesting and should be a stimulus for further research, but I do not agree that the results provide the unambiguous proof they claim. As always, replication and research focused on testing specific predictions based on the hypothesis put forward in this report, with efforts to account for alternative explanations of these observations, will be needed to determine whether the authors’ confidence in their findings is justified.

 

Posted in General | 1 Comment

Shocking News! Media Coverage of Healthcare Research Often Not Very Good.

As a veterinarian, explaining science to non-scientists and interpreting the meaning of scientific research is a key part of my job. Pet owners cannot make truly informed decisions about what to do for their animal companions without reliable information they can understand. This blog arose out of my efforts to provide better information to my clients, and it has led to further efforts to inform the public, and my colleagues in veterinary medicine, about how to evaluate medical interventions and understand the scientific research we need to support making decisions for pets.

My own knowledge about how we understand health and disease has come from many years of academic study. This includes a master’s degree I will be finishing this year in epidemiology, the branch of science specifically devoted to understanding health and disease and generating safe and effective healthcare interventions. And hopefully I have developed some ability to effectively communicate about science through my academic background, my years as a veterinarian, and my work speaking and writing for the veterinary community and, of course, in this blog.

In a sense, this blog has made me part of “The Media,” as has my involvement with the American Society of Veterinary Journalists. Unfortunately, taken as a whole “The Media” does not do a very good job of covering scientific topics, and journalists seem to contribute to misconceptions at least as often as they dispel them. A newly published study looking specifically at media coverage of healthcare research illustrates this starkly.

Schwitzer GA. A Guide to Reading Health Care News Stories. JAMA Intern Med. Published online May 05, 2014. doi:10.1001/jamainternmed.2014.1359

This paper reports on a 7-year evaluation of media stories from print and electronic media of various kinds. It details a number of specific errors in how journalists often present and interpret scientific research that lead to a false understanding of what the results mean. The conclusion of the study was

After reviewing 1889 stories (approximately 43%newspaper articles, 30% wire or news services stories, 15%online pieces [including those by broadcast and magazine companies], and 12%network television stories), the reviewers graded most stories unsatisfactory on 5 of 10 review criteria: costs, benefits, harms, quality of the evidence, and comparison of the new approach with alternatives. Drugs, medical devices, and other interventions were usually portrayed positively; potential harms were minimized, and costs were ignored.

The specific kinds of mistakes made in many stories about healthcare research struck me not only because I see them all the time in the media, but because they mirror very closely exactly the sorts of mistakes made by advocates of alternative therapies. Though this study did not, unfortunately, look specifically at coverage of alternative medicine, my subjective impression is that the media makes the same sorts of errors but is even less careful and critical in coverage of this area. Pieces on veterinary medicine, in particular, are often poor quality because they are part of the “lifestyle” or “human interest” beat rather and treated as entertainment rather than being written by qualified science journalists interested in the truth about healthcare practices.

In any case, here are the major problems the study identified in media coverage of healthcare science:

Risk Reduction Stated in Relative, Not Absolute, Terms
Stories often framed benefits in the most positive light by including statistics on the relative reduction in risk but not the absolute reduction in risk. Consequently, the potential benefits of interventions were exaggerated.

While journalists are often understandably loath to talk about anything that sounds like math, it is impossible to appropriately talk about the effects of medical therapies without identifying the difference between absolute and relative risk. If you have a 1 in a million chance of developing a terrible disease, and something raises your chances to 2 in a million, that is a relative risk increase of 100%. Sounds terrible! But the thing is, at a chance of 2 in a million, you are still almost certainly not going to get that disease. And doubling your risk does not make it meaningfully more likely that you will. Such a simple distinction is critical to deciding whether medical interventions are worthwhile

Failure to Explain the Limits of Observational Studies
Often, the stories fail to differentiate association from causation.

You may have heard the saying “correlation does not mean cause and effect.” Just because two things are associated doesn’t mean one caused the other. If, for example, a study found that carrying matches in your pocket was associated with an increase in your risk of lung cancer of ten times, would that mean matches cause lung cancer? Of course not! Carrying matches may mean you’re a smoker, and smoking certainly does cause lung cancer, but the simple association between matches and cancer doesn’t mean one causes the other.

Here’s a great site that illustrates all kinds of such bogus associations. While this may not be something everyone appreciates in daily life, journalists writing about healthcare research ought to understand it.

The Tyranny of the Anecdote
Stories may include positive patient anecdotes but omit trial dropouts, adherence problems, patient dissatisfaction, or treatment alternatives.

I’ve written about anecdotes and miracle stories many times. The number one “argument” presented in the comments on this blog in defense of treatments I evaluated critically is the presentation of anecdotes that look like they show the treatment working. Anecdotes can only suggest hypotheses to test, but they can never prove these hypotheses true.

There are many reasons treatments that don’t work may seem like they do, and professionals who interpret and explain science should know anecdotes are unreliable and often misleading. While personal stories make for more interesting and emotionally appealing narratives, they should always be used carefully only to illustrate something that has been demonstrated to be true or false by more reliable evidence.

Surrogate Markers May Not Tell the Whole Story
Journalists should distinguish changes in surrogate markers of disease from clinical endpoints, including serious disease or death. Many news stories, however, focus only on surrogate markers, as do many articles in medical journals.

The bottom line for any medical treatment is whether it reduces the meaningful symptoms of disease, including the most final of all, death. It makes no difference if a therapy raises or lowers the amount of some chemical we can measure in the blood if that isn’t a clear and well-established indicator that the therapy will also reduce suffering or prevent death. Surrogate markers are, as the article suggests, overused by healthcare researchers in many cases because they are often cheaper and easier to measure than real symptoms or mortality, but they have significant limitations, and this should be made clear when talking about research using them.

Stories About Screening Tests That Do Not Explain the Tradeoffs of Benefits and Harms
Stories about screening tests often emphasize or exaggerate potential benefits while minimizing or ignoring potential harms. We found many stories that lacked balance about screening for cardiovascular disease and screening for breast, lung, ovary, and prostate cancer.

I have frequently referred to the growing appreciation in human medicine, which has not yet come very far in the veterinary field, that screening tests have risks as well as benefits, and these need to be carefully weighed. The Choosing Wisely project is a key resource for people trying to make smart decisions about screening tests, as is the web site for the U.S. Preventative Services Task Force. Both provide real evidence to help balance the risks and benefits of potential screening tests. Journalists should be aware of the limitations and pitfalls of screening and risks such as overdiagnosis and should include those considerations in stories about screening tests.

Fawning Coverage of New Technologies
Journalists often do not question the proliferation of expensive technologies.

I would add that journalists rarely question the value or evidence for alternative therapies and tend to fawn over them and their proponents more often than not. Reporting that is truly informative and useful must be thoughtful and based on assessment of the real evidence, not simply unquestioningly enthusiastic about therapies with a token quote or two from skeptics for “balance.” Drugs are not the only medical treatment to have risks, but it seems journalists are far more likely to talk about the risks of pharmaceuticals than other treatments.

Uncritical Health Business Stories
Health business stories often provide cheerleading for local researchers and businesses, not a balanced presentation of what new information means for patients. Journalists should be more skeptical of what they are told by representatives of the health care industry.

I would argue that identifying any potential bas, financial or otherwise, in a source for a news story should be an ordinary part of journalistic practice. The idea behind seeking multiple sources is not just to provide a superficial impression of balance by including opposing points of view regardless of merit but to ensure that the journalist has a comprehensive awareness of the evidence for and against the treatment they are writing about so that they can provide a useful explanation of what is known about it. The study also found, however, that journalists often don’t follow this practice.

Single-Source Stories and Journalism Through News Releases
Half of all stories reviewed relied on a single source or failed to disclose the conflicts of interest of sources. However, journalists are expected to independently vet claims. Our project identified 121 stories (8% of all applicable stories) that apparently relied solely or largely on news releases as the source of information.

There really shouldn’t be any need to point out that this is lazy and unacceptable journalistic practice and does not lead to accurate, useful information for the public.

I don’t want to suggest that there are not many excellent journalists providing accurate and informative interpretation and analysis of healthcare research. The study specifically identifies examples of stories that succeeded in avoiding the mistakes they found, and there are certainly many in the media who do a brilliant job reporting and explaining health sciences research. Hopefully, by identifying common problems and mistakes, this study will contribute to improving the quality of healthcare science journalism.

Posted in General | 1 Comment

Evidence Update-Data for Resveratrol and Antioxidants Not Looking Good

At the very beginning of this blog in 2009, I wrote about a compound called resveratrol. I concluded at that time that it was “promising but unproven.” I subsequently reported on a scandal in which a key researcher studying this compound had a large number of papers retracted due to fraud. And last November, I passed along the conclusions of a couple of reviews of the existing evidence to that point concerning resveratrol. The conclusions had changed little from my first report. Resveratrol shows promising properties in the lab and in animal models, but it has not yet been shown to be an effective treatment or preventative health agent in humans or, of course, in pets.

A new study has recently been published which adds to the existing evidence that intake of resveratrol from natural food sources, notably red wine and chocolate, is not associated with  a reduced risk of any disease.

Semba RD. Resveratrol in red wine, chocolate, grapes not associated with improved health. JAMA Intern Med. Published online May 12, 2014. doi:10.1001/jamainternmed.2014.1582

This study followed nearly 800 people for nine years and measured the amount of resveratrol metabolites excreted in the urine, which represented how much resveratrol was being consumed in the diet. Though the study had some methodological limitations (it was an observational study, not a clinical trial, so there was no randomization, and it’s not clear from the abstract if there was any blinding for the analysis), it is a piece of data to add to our current understanding. The conclusion, unfortunately, was that

The antioxidant resveratrol found in red wine, chocolate and grapes was not associated with longevity or the incidence of cardiovascular disease, cancer and inflammation.

This does not, of course, mean that we can definitively say dietary resveratrol is of no value or that higher doses provided as a supplement might not have beneficial effects, There iss till some weak clinical trial evidence to suggest some benefits from supplementation might be beneficial in some cases. The bottom line here is that we don’t know for certain, but the compound is looking less promising the more we study it.

This is less surprising than it might once have been since the theoretical reason to expect a benefit from resveratrol is its antioxidant effects. While “antioxidant” is something of a magic word in alternative medicine, science-based medicine has been soberly and carefully investigating the idea that such compounds might have a wide variety of beneficial health effects. As I discussed earlier this year, however, the evidence is becoming more and more convincing that antioxidants have fewer benefits than once supposed and, as should be expected with all therapies that do anything at all, they come with risks.

A recent animal model study actually found evidence suggesting free radicals might in some cases be protective and that the reason they increase with age may be because they are part of the body’s attempt to fight the effects of aging rather than the cause of it. This is, of course, only one lab study, but it is a piece in a growing body of information which suggests that the hype and reality about antioxidants like resveratrol may differ quite a bit. Enthusiastic recommendations for antioxidant supplement products are clearly not warranted, and justifying therapies, including herbal and other alternative treatments, as useful by arguing they provide antioxidant effects is a weak and unconvincing rationale.

 

Posted in Herbs and Supplements | 1 Comment

Videos

This link leads to a YouTube channel with videos related to evidence-based veterinary medicine and scientific evaluation of alternative medicine.

youtubevideosnip5-2014

 

Posted in Presentations, Lectures, Publications & Interviews | Leave a comment

Finally, A Journalist Takes a Skeptical View of Claims for Veterinary Acupuncture

I am occasionally interviewed by journalists writing articles about alternative medicine for pets. Many of these articles deal focus on acupuncture, and they tend to follow a pretty consistent pattern:

1. Story about a dog or cat with some pain or disability, often that has not responded to conventional treatment.

2. Owner takes pet to veterinarian who recommends acupuncture (or other alternative treatment).

3. Brief summary of claims for benefits for acupuncture ad reference to long history of use in people and pets. Sometimes there is a reference at this point to research studies supporting the use of acupuncture (almost never to studies that do not support it).

4. Brief quote from grumpy, killjoy token skeptic (that would be me) to create the impression of “balanced” reporting.

5. Return to story of pet from beginning of article, now all better and frolicking happily.

It is well known that the media caters to our inherent preference for stories over statistics, and that when journalists cover scientific subjects, especially those that are controversial, nuance and thoughtful analysis of evidence are often sacrificed for a compelling narrative. And the well-meaning notion that a journalist should present voices from “all sides” of a controversy all too often results in stories that suggest a legitimate debate about scientific facts when, in reality, there is strong evidence and consensus on one “side” and the unshakeable faith of a small minority on the other.

I was pleased, therefore, to see a recent piece about veterinary acupuncture in Slate which took the scientific evidence, and the perspective of skeptics, more seriously than the feel-good anecdotes of believers.

If Your Veterinarian Offers Acupuncture, Find a Different Vet

Sticking needles in your dog won’t make it feel better.

By

The title may be a bit extreme, since unfortunately a lot of otherwise excellent veterinarians have been fooled by the claims and shaky evidence for acupuncture, but the overall message of the article is right on target. Mr. Palmer appreciates that while the scientific evidence is mixed, an appropriate evaluation requires considering the quality and limitations of studies and the preponderance of the evidence.

If you’re an acupuncture enthusiast, you’re probably getting ready to point me toward studies proving the efficacy of veterinary acupuncture. Before you do that, let’s make a deal: I will concede that there are studies supporting veterinary acupuncture if you concede that there are studies opposing it. The issue is assessing the quality of the studies and determining where the weight of the evidence lies.

His conclusion, with which I agree entirely, is that the most reasonable interpretation of the balance of the evidence is that acupuncture is a placebo for humans and likely has little to no predictable beneficial effects in animals. Most veterinary acupuncture studies are deeply flawed, and better research could be justified, but the best evidence in humans does not suggest this is a promising area for veterinary medicine.

Mr. Palmer’s piece also points out that the potential financial conflicts of interest which alternative medicine proponents so blithely use to dismiss research on pharmaceuticals or other conventional therapies are just as much of an issue in research on alternative veterinary therapies, including acupuncture. One of the most prominent researchers in veterinary acupuncture and so-called Traditional Chinese Veterinary Medicine (TCVM) also makes his living teaching TCVM and selling herbs and related products. While this does not automatically invalidate the research Dr. Xie is involved in, it points to a clear a priori bias which necessitates rigorous scientific controls, including replication by others, in order to generate reliable evidence. These controls are seldom present in veterinary acupuncture research.

It is encouraging to see the mainstream media identify the unimpressive scientific reality behind the widespread positive, and anecdote-driven claims for veterinary acupuncture. Here are some links to previous posts on veterinary acupuncture and TCVM which offer more details about these claims and the science, or lack of science, associated with them.

Veterinary Acupuncture

“Electroacupuncture” as a Treatment for Nausea & Vomiting Caused by Morphine in Dogs

Acupuncture vs Opioids for Surgical Pain in Dogs: Which is Better?

JAVMA Article on Electroacupuncture for IVDD

Traditional Chinese Veterinary Medicine

Evaluation of the Chinese Herbal Remedies San Ren Tang, Wei Lin Tang, and Alisma for Feline Urinary Tract Disease

The History of Veterinary Acupuncture: It’s Not What You Think

 

Posted in Acupuncture | 13 Comments