What Does “Scientifically Proven” Really Mean?

Science as a Brand
I think of science as primarily an approach to knowledge; both a philosophy and collection of methods for developing an understanding of reality. Of course science is imperfect, both in itself and in how it is practiced by imperfect human beings. But the evidence of history is overwhelming and not rationally deniable that a scientific approach to health and disease works better than any other approach tried so far.

Largely because of the dramatic successes of scientific medicine, the label of “scientifically proven” has also become a potent marketing tool. There may be suspicion of scientists and technology, and plenty of interest in vague and comforting alternative philosophies, but when the chips are down people throughout the world trust science and scientific medicine more as the most reliable source of information and healthcare.

This is why even those dedicated to ideologies clearly incompatible with science, such as homeopaths, attempt to label their practices as scientifically validated.

Because of this, it is nearly impossible to find advertising for a healthcare product or service that does not claim the product is “scientifically proven” to work. Unfortunately, most of those who use this language to market their wares do not appear to know, or to care, what it actually means, and the information to judge the truth of the claim is rarely made available to consumers.

Can We Ever Really Prove Anything?
It is often said that nothing is ever absolutely proven in science. This is true to some extent in that, contrary to anti-science propaganda, science as a method is quite open-minded and aware of the limitations of human reason, and so the method allows for the possibility that any idea, no matter how established and venerated, can be proven false with adequate evidence. However, realistically there are some propositions that achieve a level of supporting evidence that makes any serious doubt about their truth perverse and irrational. I have no fear of being wrong when I say the Earth is not flat nor poised on the back of a turtle.

Similarly, it is often said that the absence of evidence is not evidence of absence, meaning that something as yet not thoroughly tested scientifically cannot be dismissed as false, only regarded as unproven. This is certainly true as well. Unfortunately, in the marketing of unconventional healthcare products this idea is often twisted into a claim that nothing can be regarded as false without an unachievable standard of voluminous and perfect quality evidence against it. But in the real world, we can’t have perfect evidence for everything, and we must apportion our limited resources rationally.

Homeopathy, for example, has a theory that cannot be true unless much of established science is wrong; the in vitro evidence of biologic effects or water memory is overwhelmingly negative; the lab animal and human clinical trials and the few veterinary clinical trials are overwhelmingly negative, with a strong correlation between the quality of the study and the failure to find a beneficial effect. It has been exhaustively evaluated for over a century without generating compelling scientific evidence that it works. Adding all this up, it is more than reasonable to dismiss it as a failed idea and move on without insisting that without more evidence we cannot “prove” homeopathy is a waste of time.

Sorry, Reality Really is Messy
So when deciding whether or not to trust a claim that something is scientifically proven, it is important to have some idea what such a claim might actually mean, if true, and what level of evidence is necessary to reasonably make it. To begin with, the level of evidence available in medicine rarely reaches the degree of certainty of that for the hypothesis that the Earth is round and not balanced on a turtle. Therefore, reliable science-based claims for medical services should usually be circumspect and should reflect the degree of supporting data. “Scientifically proven to treat cancer with no side effects!” is not a claim that is very likely to be supportable through real evidence. “May relieve the symptoms of cancer therapy in some patients with few side effects” is far more likely to be true.

An especially frustrating conundrum for proponents of science-based medicine is that it is far easier to sell something with strong, confident, absolute claims that are probably not true than with measured, realistic assessments supported by good evidence. Ambiguity and complexity are far less attractive than simple, confident narratives. And statistics and the details of scientific evidence are less compelling to potential customers than heartwarming testimonials with far less real evidentiary value. This gives faith-based medicine a significant public relations advantage over science-based medicine and skepticism. But the fact remains that truth is messy and full of uncertainty, and if someone is selling you something as extremely simple, clear-cut, and too good to be true, they probably are selling you a hope or a fantasy, not a real medical therapy.

The Hierarchy of Evidence
The core of evidence-based medicine is the idea that not all evidence is created equal. There are levels of evidence ranging from the suggestive but not very reliable to the solid and trustworthy, though never perfect. When reading a claim that something is “scientifically proven,” you should try to find out what kind of evidence supports the claim, and make your acceptance of the claim proportional to the quality and quantity of the evidence provided.

One key principle to bear in mind is that we are by nature inclined to generate and trust evidence that confirms our beliefs. Even the smartest and most honest scientists have deep, unconscious biases in favor of their own ideas, and these biases influence how they conduct their work and how they interpret data. And such biases extend even beyond the individual. For example, it is generally far easier to publish, and publicize, studies that show a positive result than those that show a promising treatment doesn’t work. In some countries, such publication bias is so severe that negative studies are effectively never reported.

Therefore, a key principle in evaluating medical claims is Negative Evidence is More Reliable than Positive Evidence!!!! Sure, if you have ten studies by different researchers in different parts of the world, including some that deliberately set out to disprove the findings of earlier researchers, and they all get the same answer, you can be pretty sure that is the right answer. But when the evidence is conflicting, when early studies look better than later ones, when low quality studies are positive and better ones negative, or when only one research group can get positive results, the better bet by far is that the treatment being studied doesn’t work.

The following is a simplified list of kinds of evidence commonly encountered and how they ought to be interpreted. In general, the list goes from least reliable to most reliable.

1. Mere opinion– Bias and cognitive blind spots are part of being human, and they affect everybody no matter how smart or well-educated. Linus Pauling, for example, was a brilliant scientist, and arguably a really good person who won a Nobel Prize for chemistry and a Nobel Peace Prize. He was also spectacularly and obstinately wrong about the health benefits of megadoses of Vitamin C. While those who make no effort to acknowledge or control their own biases and rely entirely on faith are especially vulnerable to errors in judgment, absolutely no one is immune. So if the only evidence for a medical claim is that someone smart, kind, famous, or otherwise admirable believes it, the claim should be viewed very skeptically since this kind of evidence is deeply and persistently unreliable.

2. Testimonials– Testimonials are just stories people tell about their experiences and how they interpret them. They are not controlled in any way for bias or error, and they are highly unreliable. Testimonials are also subject to a kind of publication bias in that no one marketing a medical product or service is going to seek out, collect, and report every person’s experience. Positive stories are promoted as evidence the product works, and negative stories are ignored, suppressed, or explained away. Though the most common and emotionally compelling kind of evidence offered to sell medical products, testimonials are not scientific or trustworthy and deserve to be ignored.

3. Reasoning from First Principles– Sometimes marketers will tout something as scientifically proven and then describe some features of basic chemistry, physics, physiology, and so on to suggest how their product might work and why you should believe it does. Certainly, a theory based on established scientific principles is a lot more likely to be true than one which is not compatible with well-established knowledge. However, living organisms are complex and extrapolating from basic science to actual patients is risky and unreliable. Just because something makes sense or ought to work based on a plausible theory does not mean it actually does. This is one of those bits of evidence that is necessary but not sufficient to demonstrate a therapy is useful.

4. Test Tubes and Guinea Pigs – Basic preclinical research into a promising medical treatment begins with testing the underlying theory behind an idea. If a proposed cure for cancer doesn’t kill cancer cells in a test tube, it’s probably not worth giving to patients. But again just because something works in the simplified, controlled environment of the lab doesn’t mean it will be safe or effective in living animals. Bleach kills cancer in the test tube, but that doesn’t mean we should drink it or inject it into people or pets with cancer! Similarly, all mammals have a lot in common, so if it kills rats it could be dangerous for dogs or people, and if it cures cancer in mice, it might be useful for cancer in cats and cows. But, there are also important differences between species, and it is never appropriate to simply assume the results seen in one species will also be seen in another. This again is a type of evidence necessary to demonstrate a treatment could work, but not sufficient to claim it does.

5. Clinical Trials– This is a huge category of ways of investigating a medical treatment with many levels of reliability. Rather than describe all the different kinds of studies and their pros and cons, I will list some general factors that are useful in deciding whether the existence of a positive clinical trial justifies calling a product “scientifically proven.”

A. Strength in Numbers- Small trials are less reliable than large trials. A few trials are less reliable than many trials. Trials conducted by a couple of investigators or all in the same place are less trustworthy than trials conducted in multiple locations by many different investigators. Individual case studies or small case series are less reliable than larger group studies. In general, the credence given to clinical studies should be proportional to the size and number of the studies.

B. Controls for Bias- The more layers of control in place to minimize potential bias, the more likely the result is believable. Such controls include:

Treatment, Placebo, and No Treatment Groups
Random assignment of subject to these groups
Blinding so that as few people as possible, especially patients and those evaluating them, know who is in which group

C. Replication and Persistence- Since we all tend to see what we expect to see, even with the most honest intentions, individual studies are almost never the last word on a medical subject, especially positive studies that find what the investigators want and expect to find. A key feature to demonstrating something really works is repeated testing by different investigators in different populations. It is especially convincing when someone sets out to disprove the results of a previous study and ends up getting the same results as the original, because this requires data robust enough to contradict the biases of the researchers involved.

The Decline Effect is a phenomenon in which early studies of a new intervention look very impressive, and as further studies are done the results get less and less positive until the original effect essentially goes away. This most likely represents the scientific process gradually correcting an initial erroneous finding through attempts to replicate it. Because of this effect, single studies are rarely to be trusted as the final word on a scientific hypothesis. A true idea, a really effective therapy will withstand repeated testing and will continue to accumulate positive evidence, while a promising idea that isn’t actually true will fail to hold up to prolonged, repeated scrutiny. This is part of why systematic reviews, which are methodologically formalized surveys of the totality of the evidence provided by individual studies, are a more reliable indicator of the truth behind a claim than any of the individual studies themselves.

The Bottom Line
Because most people understand that science works better than any other method for determining which medical claims are true and which are false, promoting a therapy as “scientifically proven” is a powerful marketing tool. However, the level of evidence usually available in medicine rarely justifies sweeping, absolute claims. Less dramatic, more qualified and realistic claims are more likely to be true. And not all evidence is created equal. Some of the most common and emotionally compelling kinds of evidence used in medical advertising, such as the opinions of smart, famous people and the testimonials of satisfied customers, are actually the least reliable and most misleading kinds of evidence.

Even truly scientific evidence varies from merely suggestive to pretty clear and definitive. A plausible theory and preclinical research in test tubes and animal models are necessary to show a medical treatment could work, but they are not sufficient to prove it really does work. And even evidence from published clinical trials must be given credence in proportion to the size, number, and quality of the studies that support a claim. Finally, since our biases are almost always in favor of confirming what we already believe to be true, negative evidence is far more reliable than positive evidence.

Share on Facebook

Tweet

Follow us