Evidence-Based Medicine: Separating the Wheat from the Chaff |

As I have discussed before, evidence-based medicine is the formal, explicit application of the philosophy and methods of science to generating understanding and making decisions in veterinary medicine. Science shows us how the world works, and in medicine it shows us which of our therapies are effective. But perhaps even more importantly, it shows us which are ineffective or not safe. A key to practicing truly science-based medicine is accepting that our perceptions and appearances can be deceiving, and often we will be wrong in our assessment of which therapies work and which don’t. This means we must have the humility and courage to give up cherished beliefs and practices when science clearly shows us our faith in them is wrong.

This is one of the key differences between science-based and alternative medicine. The very category of complementary and alternative medicine (CAM) exists precisely to protect from scrutiny and criticism those therapies that have either been shown not to work or that haven’t been adequately tested. As I have discussed previously, CAM proponents often have a fundamental philosophical distrust of science and the hierarchy of evidence, and they often believe personal experience, intuition, tradition, and other low-reliability Potential considerations in prioritizing the testing of unproven medical practices.kinds of evidence are as good as or superior to scientific research. There is a general human tendency to resist change and to cling to beliefs and habits even when there is good reason to abandon them, but this is an even greater problem in CAM than in conventional medicine because it is part of the philosophical rationale of alternative medicine.

A recent paper by John Ioannidis, one of the most productive critics of the scientific literature and process, emphasizes the natural and important role of rejecting the disproven and targeting our research efforts to most effectively and efficiently separate the safe and effective from the unsafe and ineffective.

Vinay Prasad and John PA Ioannidis. Evidence-based de-implementation for contradicted, unproven, and aspiring healthcare practices. Implementation Science 2014, 9:1

In this paper, the authors illustrate how identifying ineffective or unsafe practices, and contradicting current practices, is a common and intrinsic part of the process of science-based medicine. There are hundreds examples of such practices being identified and abandoned, just as there are examples of new ideas being treated skeptically and then accepted once adequate supportive evidence is developed. It would be difficult to demonstrate a similar pattern for most alternative disciplines, where new ideas are often accepted on theoretical or anecdotal grounds alone, and where few ideas are rigorously tested or rejected even when negative evidence is available.

The authors also discuss how the natural human resistance to change plays out and affects both the scientific literature and the response to evidence about existing practices. The purpose of the article is specifically to develop and advocate for a strategy to efficiently use scientific research to change medical practices. Such a strategy would not be necessary if people naturally made decisions in a rational, disinterested, and evidence-based way. However, the difference between science-based medicine and alternative medicine lies largely in the fact that compensating for our innate cognitive biases is built into the very fabric of science, whereas following the guidance of these biases is the favored approach in much of the alternative medical community.

The authors also review the issue of how truly evidence-based conventional medical practices are. This is often raised by defenders of alternative medicine with the goal of invoking the tu quoque fallacy, that proponents of science-based medicine cannot legitimately criticize the poor evidence base for alternative medicine while scientific medicine has an imperfect evidentiary foundation. The authors do identify significant deficiencies in the evidence base of conventional medicine, reporting for example that a review of Cochrane systematic reviews found 49% of interventions could not be definitively validated or contradicted based on the evidence, and 48% of recommendations from the American Academy of Cardiology were based only on expert opinion. This is still quite a bit better than the claims made by proponents of alternative medicine, and there is reason to believe that when all forms of scientific evidence are considered, most conventional medicine has a significantly more solid foundation in evidence than the bulk of alternative medicine. Nevertheless, it is clear that there is significant room for improvement in the degree to which conventional medicine is evidence-based.

This paper also addresses the issue of how to prioritize research on unproven therapies. While it doesn’t address alternative medicine directly, and so doesn’t deal with the issue of prior plausibility, it does lay out an interesting approach. While there is room for debate about the specifics, I appreciate the general approach. It is not appropriate to say that before rejecting any therapy we must always have high-quality and high-level evidence. The resources for scientific studies are limited, and we cannot test every approach no matter how unlikely to be useful simply because someone believes in it. And there are ethical problems with subjecting people or animals to studies of therapies that are irrational or not adequately evaluated at the preclinical level, as demonstrated by the awful experience of patients with pancreatic cancer treated with the Gonazalez Regime. Deciding how much effort to put into testing specific unproven therapies is a complex balance of different considerations, and this paper addresses several of these.


Potential considerations in prioritizing the testing of unproven medical practices
Factor to consider	General principle	How to implement this factor
Prior evidence base	Priority should be given to practices where the present evidence base is weakest.	For instance, a tiered system may be utilized: Level 1 (Weak) Randomized trials of interventions claiming subjective benefits, that are unblinded or fail to use proper controls. 2 (Weaker) Historically controlled studies of interventions that purport survival benefits, case series documenting improvements in subjective endpoints and quasi-experimental studies. 3 (Weakest) Practices based on pathophysiology and expert opinion alone. In many cases, professional conflicts may also prove problematic; thus, it may be reasonable to pursue this technique using content-specific experts in strictly an advisory capacity
Cost/ubiquity	Priority should be given to interventions with significant net financial burden on health payers.	For instance, orthopedic procedures for chronic back and joint pain, including knee and hip replacement surgeries are widely utilized in the United States, incur large financial burden on payers, but have little evidence of sustained long term benefits.
Alternative options	Priority should be given to practices for which there are several alternative options, particularly if alternatives are of completely different mechanisms (thus unlikely to also be overturned), or of low cost or bolstered by stronger evidence.	For instance, consider the market for anti-rheumatologic agents. Maintenance treatment of rheumatoid arthritis (RA) with disease modifying agents (DMARDS) has historically relied upon oral anti-immunologic agents such as methotrexate, azathiaprine, cyclosporin, and hydroxychloroquine. Recent years have witnessed a boom in novel drugs, typically expensive monoclonal antibodies against circulating cytokines or cell surface receptors. To date, this market has been limited by paucity of head to head trials, and, of trials that have been conducted, the majority are industry-sponsored studies. Collectively, there remains clinical uncertainty about how best to use these agents [20].
Documented harms	Priority should be given to test practices where the harms are well documented and confer substantial morbidity.	For instance, there is growing awareness of strut fracturization, embolism, and migration of IVC filters. At the same time, the IVC filter has never shown to improve any patient-centered outcome for any patient population in a prospective trial, and traces its approval through the FDAs 510 k mechanism [21].
Testing the intervention makes financial sense	Priority should be given to test practices where the cost to test is far less than ongoing expenditures of the practice.	In some respects, trialists should think like CEOs, weighing the costs of conducting a study, which may find a practice ineffective versus the ongoing expenditures for that practice. At times, such calculations may favor costly trials where the existing evidence base is weak, observational studies suggest inefficiencies, and the ongoing costs are large [22]. At other times, small trials that eliminate boutique practices may be employed [23]. Whose financial bottom line is being affected is important to consider. For that reason, nonconflicted bodies should make these determinations, utilizing investigators without financial conflicts of interest.
Proponents are open-minded	Priority should be given to test practices where negative results may truly gain traction.	Some specialties (primary care providers) may be more ready to abandon contradicted medical practices, and it is reasonable to test practices when there is genuine belief that contradiction can gain traction. Furthermore, some practices may be cumbersome (tight glycemic control in the ICU), time-consuming (routine gown and glove precautions) or unpleasant, and their contradiction may also be palatable. Finally, as payment structures shift from fee for service towards bundles [24], costly components may lose faithful disciples. Other fields, those with numerous and hyperbolic third party advocates, have been notoriously unwilling to trust results that undermine their worldview, no matter how robust the science.
Value of information gained	Priority should be based on the expected value of funding a specific study that may inform de-implementation, at the size and cost proposed.	Value of information (VOI) offers a decision-making framework that tries to capture several of the above issues, at least the ones that can be best quantified [19]. VOI can be used to prioritize and power clinical trials taking into account the costs of increasing study sample size, the potential number of persons affected by changes in that practice, the costs of the practice, including downstream costs, and the increased knowledge of marginal changes in health outcomes that may result from testing — converting all to the final common denominator of cost per favorable outcome gained.

I particularly appreciate the consideration of the impact of the evidence gained from research. If negative results are not going to have any impact, is research really justified? Clearly, we do not need more research on homeopathy for many reasons, for example, but a big reason is that additional research will never dissuade true believers but will simply add to the illusion that there is some scientific legitimacy to the ideas or practices they employ. Likewise, the money spent through the National Center for Complementary and Alternative Medicine has failed to validate most of the therapies studied, yet it has not apparently led to the rejection of any therapies despite the negative evidence generated.

Skeptics like myself are often accused of blind, unthinking defense of conventional practices and closed-mindedness. The exact opposite, however, is the truth about skepticism. It is the process of demanding evidence for all practices, including those I personally rely on. Articles like this illustrate that truly science and evidence-based medicine is as rigorous and critical in approaching its own practices as those under the label of CAM. This kind of critical scrutiny and adherence to the evidence, whatever it reveals, is necessary for progress and improvement in medicine, and is the acceptance of that that distinguishes science-based medicine from alternative approaches to developing and employing medical knowledge.