Any time there is discussion of alternative medicine, of treatments that haven’t passed the test of rigorous scientific investigation (or even been tested at all), there are miracle stories. Testimonials, anecdotes, whatever you call them, they are tales of amazing recoveries from illness when everything tried had failed and all hope was lost. The comments on this blog contain many such stories for almost all of the products and methods I have questioned.
And such stories provide emotionally compelling evidence for the effectiveness of these interventions. If an owner has tried a hundred treatments for their pet, or been told there is no hope and nothing can save the animal, and then they try one more thing and their companion recovers, surely this means that final treatment must have worked? And yet, with all these testaments to the power of these alternative methods, I still have doubts. Let’s take a look at why that is.
There are a number of arguments against believing that testimonials prove anything about the effects of a medical treatment. Most of these, of course, will have no effect on the individual experiencing the miracle. It is in the nature of human beings to find their own experiences, and stories from others, deeply compelling, and sadly facts and logical arguments have little impact on that fundamentally emotional and irrational sense of certainty. Still, making the effort to understand the limitations of anecdotes has led us to this point in time, a point in which our medicine is more effective by far than any time in human history. So there is clearly value in understanding the problem of miracle stories even if many people will still find them compelling.
Things Aren’t Always What They Seem
The first classic argument is simply that things aren’t always what they seem. There are many reasons why the fact that a patient gets better after receiving a particular medical treatment doesn’t actually tell us the treatment made the patient better. The notion that if an action preceding an outcome means the action must have caused that outcome is known as the post hoc ergo propter hoc fallacy. In medicine, there is a long list of reasons why this kind of reasoning about causation isn’t trustworthy:
- Self-limiting Disease:
Most medical problems ultimately go away by themselves. Certainly minor ones nearly always do, and even major, life-threatening diseases like cancer resolve spontaneously despite our belief that they can’t or shouldn’t. It is natural to assume that whatever we did last, right before the patient got better, is the reason they got better. Natural, but not reliably true.
- Regression to the Mean:
Chronic diseases, especially mysterious ones we don’t fully understand, tend to come and go. In addition to spontaneously going away altogether, which they can even after many years, they commonly tend to get better for a while then worse for a while and back and forth in an unpredictable cycle which we are always trying to understand and predict. Some stories of miraculous results from medical treatments are simply cases in which the patient was having a particularly bad run of symptoms and then felt better after getting treated.
If we systematically followed up all these stories, we would find a fair number of people whose symptoms returned despite continuing the supposedly effective treatment. Just like we find that almost any fad diet works for some people, and yet by a year most of them have gained back the weight they lost initially. Companies and practitioners selling alternative therapies, however, don’t make a point of systematically following up on all their satisfied customers’ testimonials to see which eventually got sick again. I wonder why that is?
- Placebo Effect:
Everyone hates this one because it means you believe something which isn’t actually true. You use a treatment and feel better, but if it’s just a placebo and your disease isn’t actually any different, that means you must be crazy or stupid, right? No! Claiming that a perceived change is only the result of a placebo effect isn’t an insult to the patient. It is simply a fact that all human beings see want we want to see a fair portion of the time, and that sometimes we interpret what we see to mean something that it doesn’t really mean.
In the case of medical therapies, almost anything I give a client, they will perceive some improvement right after simply because we’ve done something and we know we are supposed to be looking hard for any sign of an effect, especially since we really want there to be one. Again, systematically following up all these early responses often reveals no meaningful change, but if you ask at the right time, the answer will almost always be positive and suitable for use as a testimonial.
- Misdiagnosis:
Sometimes, the reason a disease went away after an unconventional treatment (or a conventional one, for that matter) was used is because the disease was never there to begin with. Doctors aren’t perfect, and they sometimes diagnose diseases the patient doesn’t have. Any certainly people who aren’t trained medical professionals but who diagnose themselves or their pets are quite likely wrong a lot of the time. It’s easy to cure a disease that a patient didn’t have, so one of the first questions to ask about any miracle story is what hard, objective, tangible evidence is there to support the diagnosis? If your dog had cancer and it was cured by some kind of pseudoscientific nonsense like homeopathy, there had better be extensive imaging and before and after biopsies with long-term follow up before you start claiming homeopathy actually works and all the laws of physics and chemistry are mistakes.
You Can’t Prove It Isn’t a Miracle!
Another problem with miracle stories as proof that specific medical treatments actually work is that such stories can’t ever be proven wrong. And if something can’t be proven wrong no matter what, then you have to believe each and every one equally, which as we will see is probably the biggest problem with anecdotes.
Of course, the main reason you can’t disprove a miracle tale is psychological; cognitive dissonance is such a powerful and ubiquitous feature of the human brain, that it is almost impossible to convince someone they are wrong about a strongly emotional personal experience. Just as it is a great relief and comfort to know that you saved your beloved pet by using an unconventional therapy your vet said wouldn’t work, it is equally terrifying to believe that actually the therapy had nothing to do with your pet getting better and that not only did you waste your money and put your pet at unnecessary risk, but the world is ultimately unpredictable and uncontrollable and “As flies to wanton boys, are we to the gods. They kill us for their sport.” (King Lear) People cling to their miracles largely because these stories give them hope that bad things can be avoided or controlled if we just find the right things to do or believe.
Another reason miracle stories are unfalsifiable, which makes them largely useless as a form of evidence in science, is that they consist of very limited and selective information. By definition, anecdotes presented to show a treatment works contain only information that supports the effectiveness of the treatment. It wouldn’t be a miracle if at the end of the story the disease killed the patient. But does that mean there aren’t stories about the same treatment in which there was no benefit? Or even great harm? No, of course not. Testimonials are a form of “evidence” that is positive by definition, and that necessarily excludes negative evidence no matter how much of it might actually exist.
Of course, for a true believer, it would make no difference even if one could produce a story of failure for every single story of success. Cognitive dissonance would simply move the believer to say, “Well, maybe it doesn’t work for everybody, but it sure worked for me!” But for anyone trying to make an objective decision about how likely a particular therapy is to be effective, it is worth noting that testimonials and other anecdotes leave out all the stories that show the therapy failing, so they are an inherently biased kind of evidence.
Everything Works!
Though it doesn’t always show up on the typical lists of reasons why we shouldn’t trust miracle stories, to me the most damning argument against them is simply that they can be found supporting every medical treatment ever invented. On this blog alone, you can find a testimonial in the comments for nearly every product or therapy I have questioned, despite the fact that most of them have entirely different theories and methods by which they are supposed to work. And if you look at alternative medicine in general, there is not a single practice that you can’t find lots of people with powerful, emotional stories showing positive results.
But let’s go beyond the alternative therapies popular in the US today. Sure, there are plenty of testimonials for, say, Double Helix Water. How about for other kinds of magic water?
Homeopathy (after all, it’s just water)
Lourdes Water
Electron Water
pH Miracle Water
Kangen Water
Hexagon Water
Pranic Water
Ypsilanti Water
And even just Plain Water
So does every kind of water cure disease? Maybe the problem isn’t that testimonials aren’t unreliable. Maybe we’ve just stumbled across the fact that water itself is medicine? Ah, but I wonder then if we can find the same testimonials for every other therapy ever invented? Let’s see.
Christian Spiritual Healing
Islamic Spiritual Healing
Hindu Spiritual Healing
Christian Science Healing
Ancient Greek Healing
Ancient Roman Healing
Norse Magical Healing
Shamanic Healing
Theta Energy Healing
Ritual Child Sacrifice for Healing
Crystal Healing
Laser Healing
Bloodletting Healing
Magnetic Healing
Angel Therapy
Urine Therapy
Anyway, this list could be endless even without including the most popular alternative therapies. But it seems to suggest we should ask this question: If testimonials and miracle stories exist to support every medical treatment ever invented, does that mean every treatment works, or does it mean testimonials aren’t reliable?
Testimonials are useful only to generate hypotheses, or to illustrate theories that must be demonstrated by other means. By themselves, they prove nothing. Not only are they not reliable for all the reasons I’ve discussed, but if we accept them as compelling evidence, we commit ourselves to believing in everything at once, or simply arbitrarily choosing which stories to believe and which to doubt. This does not lead us to separating truth from error.
Pingback: RenAvast™ for Kidney Disease: Sloppy Science and Snake Oil Marketing | The SkeptVet Blog
Pingback: Tumexal: Wonder Drug or Snake Oil? | The SkeptVet Blog
The view on placebo effect is incomplete. Placebo effects (as well as Nocebo effects) can be real and can be the result of actual physiological changes that take place in the body as a result of some physical activity taken beforehand. They are not just purely “imaginary” or just “wishful thinking” or “magical thinking” even though, at times, they could be. There are measured effects with acupuncture, even fake acupuncture, and any particular “placebo treatment” has the potential, via the power of the mind on the body’s processes, to make alterations in the body’s biochemistry and physiology. So the Placebo Effect cannot be dismissed in the way the author suggests.
You’ve missed the point. Anything that affectes our thinking is, of couarse, a physical effect since our thinking is a function of physical things that happen in our brain. But you are confusing this with a predictable, meaningful change in the nature of physical health or disease.
Acupuncture is a great example. It causes physical effects (changes in neurotransmitters, blood pressure, heart rate, etc), and it causes the perception of improvement. But when the disease can be objectively measured physically, it does not cause consistent, predictable changes in that. One study, for example, found that people who had acupuncture treatment for asthma felt they could breathe better. This sounds like a good thing until you get t the part of the study where their lung function was unchanged. That means people are likely to give up on therapies that actually make their lungs better because they feel that acupuncture has helped them when the effect has been on their perceptions but not on their lungs. This may be a benefit in terms of subjective symptoms, but it leaves the disease itself untreated. Real medicine, on the other hand, offers the same subjective symptom benefits through the placebo effect and treats the disease as well. This is a better choice for the patient.
Pingback: Shocking News! Media Coverage of Healthcare Research Often Not Very Good. | The SkeptVet Blog
Pingback: Testimonials Lie: More Evidence for Why You Can’t Trust Anecdotes or Personal Internet Reviews of Medical Treatments | The SkeptVet
Pingback: How to Pick a Vet (or How Not To) | The SkeptVet
Pingback: Don’t Believe Your Eyes (or Your Brain) | The SkeptVet
Pingback: Update: RenAvast Banned by the FDA Due to Illegal Claims | The SkeptVet
“Testimonials are a form of “evidence” that is positive by definition, and that necessarily excludes negative evidence no matter how much of it might actually exist.”
No they’re not.