Evidence-Based Medicine: Separating the Wheat from the Chaff

As I have discussed before, evidence-based medicine is the formal, explicit application of the philosophy and methods of science to generating understanding and making decisions in veterinary medicine. Science shows us how the world works, and in medicine it shows us which of our therapies are effective. But perhaps even more importantly, it shows us which are ineffective or not safe. A key to practicing truly science-based medicine is accepting that our perceptions and appearances can be deceiving, and often we will be wrong in our assessment of which therapies work and which don’t. This means we must have the humility and courage to give up cherished beliefs and practices when science clearly shows us our faith in them is wrong.

This is one of the key differences between science-based and alternative medicine. The very category of complementary and alternative medicine (CAM) exists precisely to protect from scrutiny and criticism those therapies that have either been shown not to work or that haven’t been adequately tested. As I have discussed previously, CAM proponents often have a fundamental philosophical distrust of science and the hierarchy of evidence, and they often believe personal experience, intuition, tradition, and other low-reliability Potential considerations in prioritizing the testing of unproven medical practices.kinds of evidence are as good as or superior to scientific research. There is a general human tendency to resist change and to cling to beliefs and habits even when there is good reason to abandon them, but this is an even greater problem in CAM than in conventional medicine because it is part of the philosophical rationale of alternative medicine.

A recent paper by John Ioannidis, one of the most productive critics of the scientific literature and process, emphasizes the natural and important role of rejecting the disproven and targeting our research efforts to most effectively and efficiently separate the safe and effective from the unsafe and ineffective.

Vinay Prasad and John PA Ioannidis. Evidence-based de-implementation for contradicted, unproven, and aspiring healthcare practices. Implementation Science 2014, 9:1

In this paper, the authors illustrate how identifying ineffective or unsafe practices, and contradicting current practices, is a common and intrinsic part of the process of science-based medicine. There are hundreds examples of such practices being identified and abandoned, just as there are examples of new ideas being treated skeptically and then accepted once adequate supportive evidence is developed. It would be difficult to demonstrate a similar pattern for most alternative disciplines, where new ideas are often accepted on theoretical or anecdotal grounds alone, and where few ideas are rigorously tested or rejected even when negative evidence is available.

The authors also discuss how the natural human resistance to change plays out and affects both the scientific literature and the response to evidence about existing practices. The purpose of the article is specifically to develop and advocate for a strategy to efficiently use scientific research to change medical practices. Such a strategy would not be necessary if people naturally made decisions in a rational, disinterested, and evidence-based way. However, the difference between science-based medicine and alternative medicine lies largely in the fact that compensating for our innate cognitive biases is built into the very fabric of science, whereas following the guidance of these biases is the favored approach in much of the alternative medical community.

The authors also review the issue of how truly evidence-based conventional medical practices are. This is often raised by defenders of alternative medicine with the goal of invoking the tu quoque fallacy, that proponents of science-based medicine cannot legitimately criticize the poor evidence base for alternative medicine while scientific medicine has an imperfect evidentiary foundation. The authors do identify significant deficiencies in the evidence base of conventional medicine, reporting for example that a review of Cochrane systematic reviews found 49% of interventions could not be definitively validated or contradicted based on the evidence, and 48% of recommendations from the American Academy of Cardiology were based only on expert opinion. This is still quite a bit better than the claims made by proponents of alternative medicine, and there is reason to believe that when all forms of scientific evidence are considered, most conventional medicine has a significantly more solid foundation in evidence than the bulk of alternative medicine. Nevertheless, it is clear that there is significant room for improvement in the degree to which conventional medicine is evidence-based.

This paper also addresses the issue of how to prioritize research on unproven therapies. While it doesn’t address alternative medicine directly, and so doesn’t deal with the issue of prior plausibility, it does lay out an interesting approach. While there is room for debate about the specifics, I appreciate the general approach. It is not appropriate to say that before rejecting any therapy we must always have high-quality and high-level evidence. The resources for scientific studies are limited, and we cannot test every approach no matter how unlikely to be useful simply because someone believes in it. And there are ethical problems with subjecting people or animals to studies of therapies that are irrational or not adequately evaluated at the preclinical level, as demonstrated by the awful experience of patients with pancreatic cancer treated with the Gonazalez Regime. Deciding how much effort to put into testing specific unproven therapies is a complex balance of different considerations, and this paper addresses several of these.

Potential considerations in prioritizing the testing of    unproven medical practices

Factor    to consider

General principle How to implement this factor

Prior   evidence base

Priority   should be given to practices where the present evidence base is weakest.

For   instance, a tiered system may be utilized: Level 1 (Weak) Randomized trials   of interventions claiming subjective benefits, that are unblinded or fail to   use proper controls. 2 (Weaker) Historically controlled studies of   interventions that purport survival benefits, case series documenting   improvements in subjective endpoints and quasi-experimental studies. 3   (Weakest) Practices based on pathophysiology and expert opinion alone. In   many cases, professional conflicts may also prove problematic; thus, it may   be reasonable to pursue this technique using content-specific experts in   strictly an advisory capacity


Priority   should be given to interventions with significant net financial burden on   health payers.

For   instance, orthopedic procedures for chronic back and joint pain, including   knee and hip replacement surgeries are widely utilized in the United States,   incur large financial burden on payers, but have little evidence of sustained   long term benefits.

Alternative   options

Priority   should be given to practices for which there are several alternative options,   particularly if alternatives are of completely different mechanisms (thus   unlikely to also be overturned), or of low cost or bolstered by stronger   evidence.

For   instance, consider the market for anti-rheumatologic agents. Maintenance   treatment of rheumatoid arthritis (RA) with disease modifying agents (DMARDS)   has historically relied upon oral anti-immunologic agents such as   methotrexate, azathiaprine, cyclosporin, and hydroxychloroquine. Recent years   have witnessed a boom in novel drugs, typically expensive monoclonal   antibodies against circulating cytokines or cell surface receptors. To date,   this market has been limited by paucity of head to head trials, and, of   trials that have been conducted, the majority are industry-sponsored studies.   Collectively, there remains clinical uncertainty about how best to use these   agents [20].

Documented   harms

Priority   should be given to test practices where the harms are well documented and   confer substantial morbidity.

For   instance, there is growing awareness of strut fracturization, embolism, and   migration of IVC filters. At the same time, the IVC filter has never shown to   improve any patient-centered outcome for any patient population in a   prospective trial, and traces its approval through the FDAs 510 k   mechanism [21].

Testing   the intervention makes financial sense

Priority   should be given to test practices where the cost to test is far less than   ongoing expenditures of the practice.

In   some respects, trialists should think like CEOs, weighing the costs of   conducting a study, which may find a practice ineffective versus the ongoing   expenditures for that practice. At times, such calculations may favor costly   trials where the existing evidence base is weak, observational studies   suggest inefficiencies, and the ongoing costs are large [22]. At   other times, small trials that eliminate boutique practices may be employed [23]. Whose   financial bottom line is being affected is important to consider. For that   reason, nonconflicted bodies should make these determinations, utilizing   investigators without financial conflicts of interest.

Proponents   are open-minded

Priority   should be given to test practices where negative results may truly gain   traction.

Some   specialties (primary care providers) may be more ready to abandon   contradicted medical practices, and it is reasonable to test practices when   there is genuine belief that contradiction can gain traction. Furthermore,   some practices may be cumbersome (tight glycemic control in the ICU),   time-consuming (routine gown and glove precautions) or unpleasant, and their   contradiction may also be palatable. Finally, as payment structures shift   from fee for service towards bundles [24], costly   components may lose faithful disciples. Other fields, those with numerous and   hyperbolic third party advocates, have been notoriously unwilling to trust   results that undermine their worldview, no matter how robust the science.

Value   of information gained

Priority   should be based on the expected value of funding a specific study that may   inform de-implementation, at the size and cost proposed.

Value   of information (VOI) offers a decision-making framework that tries to capture   several of the above issues, at least the ones that can be best quantified [19]. VOI   can be used to prioritize and power clinical trials taking into account the   costs of increasing study sample size, the potential number of persons   affected by changes in that practice, the costs of the practice, including   downstream costs, and the increased knowledge of marginal changes in health   outcomes that may result from testing — converting all to the final common   denominator of cost per favorable outcome gained.

I particularly appreciate the consideration of the impact of the evidence gained from research. If negative results are not going to have any impact, is research really justified? Clearly, we do not need more research on homeopathy for many reasons, for example, but a big reason is that additional research will never dissuade true believers but will simply add to the illusion that there is some scientific legitimacy to the ideas or practices they employ. Likewise, the money spent through the National Center for Complementary and Alternative Medicine has failed to validate most of the therapies studied, yet it has not apparently led to the rejection of any therapies despite the negative evidence generated.

Skeptics like myself are often accused of blind, unthinking defense of conventional practices and closed-mindedness. The exact opposite, however, is the truth about skepticism. It is the process of demanding evidence for all practices, including those I personally rely on. Articles like this illustrate that truly science and evidence-based medicine is as rigorous and critical in approaching its own practices as those under the label of CAM. This kind of critical scrutiny and adherence to the evidence, whatever it reveals, is necessary for progress and improvement in medicine, and is the acceptance of that that distinguishes science-based medicine from alternative approaches to developing and employing medical knowledge.



This entry was posted in Science-Based Veterinary Medicine. Bookmark the permalink.

One Response to Evidence-Based Medicine: Separating the Wheat from the Chaff

  1. Pingback: Falconer Strikes Back: A Veterinary Homeopath Refutes the Skeptics and Reveals the Emperor’s Nakedness | The SkeptVet Blog

Leave a Reply

Your email address will not be published. Required fields are marked *