By far the most frequent response I get to any article critical of claims for an alternative therapy or focused on the lack of good evidence for such claims is an appeal to anecdotal evidence. Some variation on “I tried it and it worked for me” or “Don’t knock it ’til you’ve tried it” is a common and, for many people, persuasive rebuttal to even the most carefully researched and objective, science-based critique. Since I end up responding to such anecdote-based rebuttals almost every day, I have written frequently about why anecdotes and testimonials aren’t reliable evidence and why personal experience isn’t a necessary, or even very good, way to evaluate a medical treatment. I am collecting these articles here for convenience, both as a reference for interested and open-minded readers and so I only have to post one link when I answer such arguments.
Why We’re Often Wrong
Testimonials Lie
The Role of Anecdotes in Science-Based Medicine
Don’t Believe your Eyes (or Your Brain)
Caregiver Placebo Effects
Though a bit more technical and aimed at vets, this article also touches on cognitive biases which reduce the reliability of uncontrolled personal observations.
McKenzie, B. Veterinary clinical decision-making: cognitive biases, external constraints, and strategies for improvement. Journal of the American Veterinary Medical Association. 2014;244(3):271-276.
I get this error on the first link.
Fatal error: Incompatible file format: The encoded file has format major ID 1, whereas the Loader expects 7 in /home/skeptve1/public_html/index.php on line 0
If you type “Why we’re often wrong” in the search engine on the right above recent comments, I think it will take you to the blog.
Sorry about the link. I’ve fixed it and checked all the others. Thanks!
my ten year old Rottweiler has bone cancer. I give her the prescribed meds and essiac because it was recommended to me and “can’t hurt” right? She did a downward spiral so abnormally fast with all around joint pain I thought I was loosing her 3 weeks after diagnosis. On a whim I stopped the essiac. She has improved every day since. It has been 5 days no essiac she is back to a slight limp and running around coincidence? I can’t find any info on this.
Unfortunately, such anecdotes aren’t any better at telling us what doesn’t work than they are at telling us what does. We could line up a thousand such stories, for or against the treatment, and we still wouldn’t have a reliable answer. That’s why science is needed.
My dog was put on a Chinese herb because of a cancerous tumor on his leg. It was removed and got all of it but was suggested to prevent it returning. It has been 11/2 yrs now and he just had a 4 day hoist if seusres and now on Levetiracetam and seizures stopped but I now want to end the herbs and try cbd oil to try to get him off of antiseizure meds along with adding cooked good food w calcium, salmon oil with reputable dog food may change health a good direction for adding a few more years to his life. I cannot afford a mri for a pinpoint on why the seizures. The vet is just clinically giving him what will immediately help. I wonder if the herb he was living on just had a reaction? I am giving him xue fu Zhu Tang 1/2 tsp 1x day. Along with 25 mg benadryl and 10 mg Famotadine and Ligaplex 11 for joints? Too much?
There is zero clinical research evidence to support the safety or effectiveness of Chinese herbal remedies, and there have been many reports of contamination with dangerous toxins or unreported drugs, so I do not recommend using them. While CBD has been shown to help human children with a specific seizure disorder, it is not yet clear if ithelpsdogsacd, unfortunately, the cause does matter in terms of choosing the remedy.Unlike the TCM herbs, there is less reason to be concerned about serious side-effects from the CBD. Good luck.
Der Skeptvet, I would like to share my thoughts with you.
I am not opposed to science in any way, but the problem that I see is that for everything we think we know, there is so much more we don’t. I know that you acknowlege this. But from my personal experience both with human doctors and veterinarians, this is not always the case.
What I have found in the traditional medical field is that curiosity is often lacking. Of course I can only speak from personal experience, and experience of those close to me, and I can’t tell you a “percentage” of medical practitioners this applies to.
But, I truly believe one of the main problems in the medical field, and also something that drives people away from traditional science-based medicine, is the lack of curiousity ie willingness to further knowledge beyond what is minimally neccesary.
It is fully possible to treat a patient based on science that was valid 15 years ago, completely disregarding, or not even knowing about the scientific advances that have been made the latest 15 years. If this approach is used, it is not truly science-based.
I have a dog with Cushing’s disease and have made several ACTH-stimulation-tests and full blood tests.
According to best practice, the blood tests should be made on an empty stomach.
According to best practice, the ACTH-stimulation-test should be done after the dog has been fed and medicated as usual.
My veterinarians choice was to perform the blood tests and the ACTH-test at the same time, on a fasted stomach, although the medicine could be given the same morning.
The manufacturer of the medicine (Vetoryl) claims that it is very important that the dog is fed the same morning before the test is done, my veterinarian went against this.
I called my veterinarian and asked about this, he told me that it is fine and that I should trust him because he goes to conferences etc and knows what he is doing.
In many aspects, he is a competent veterinarian, he is the one who made me aware that the new, quick-test for measuring cortisol is not as reliable and it all made sense
He is also the one who adviced me to give my dog the medicine with a little bit of fat for increased absorption which makes sense because it is a fat soluble.
However, in this particular instance, I do not trust his judgement. I told him that the manufaturer stresses the importance that nothing changes in the routine on the day of the test, that it is given with food as usual etc. but he would not have it.
He insisted that I need to bring my dog fasted becase if I don’t, the blood values will not be accurate (liver, cholesterol etc).
So I suggested that I do the ACTH-stimulation-test and the blood test on two separate occations.
He did not want to do this because he thought it would add unneccesary stress for my dog.
I can not accept this reason.
The stress is a problem to begin with since it elevates cortisol, so simply going to the veterinarian may already give him a false result.
The correct result is extremely important since the medication dosage is dependend on it. Giving the medication without food may affect how the medicine is absorbed.
And not overdosing is extremely important because too much can send the dog into an adrenal crisis.
In my humble, uneducated opinion, going against the manufacturers advice, and changing the routine on the day of the test is not a safe thing to do, no matter how many seminars you have gone to, or how many years of experience you have.
The veterinarian could not provide me an appropriate explanation as to why doing it his way was safe, but instead asked me to trust him.
And this veterinarian was still curteous to engage in conversation with me, acknowledged that I had done a lot of research and did not dismiss me entirely.
Still, this did hurt my trust in him as a medical provider for my dog.
Many veterinarians and doctors unfortunately are very confident in their own knowledge to the point that they can come off as lacking humility.
And we are all only human, we are all wrong all the time so humility is very important.
And unfortunately, this is the kind of experience that many people have with medical professionals that increases scepticism, decreases compliance, and in too many cases, sends the person straight into the claws of an “alternative doctor” that now confirms what they have now come to believe about the medical field, and offers them a plausible cure-it-all.
I understand that it is frustrating to deal with false accusations and people believing in miracle cures, I understand that it gets old, but oftentimes, these people are people who feel let down by the medical professionals, either because of actual malpractice in some cases, or because of lack of communication.
I would like to say that again, the most important factor or lack of communication.
You need to understand that even though a regular person who lacks a medical education can only get fragments of the knowledge you have aquired from years of studying and practicing, it is still important to acknowledge that the every day person now has access to more information than ever before, and it is very important, if you are to dismiss a persons claims, to explain why they are wrong, but also to be open to the idea that perhaps they have gotten access to information that you have not yet.
With all that is going on in all things science, it is impossible for one single individual to be flawlessly up to date with each and every aspect of medicine.
Sorry for the long text, I hope it is not too incoherent.
In general, I agree with you. Communication is critical, and not all healthcare workers are good at it, just as not all people in other professions are. We are, after, just people like anyone else, with strengths and weaknesses. I also agree that sometimes individual doctors, just like pet owners, trust their personal experiences more than scientific evidence and so make choices that contradict the current best evidence. That is precisely the sort of problem evidence-based medicine attempts to solve, by encouraging us to trust the evidence more than our own beliefs, opinions, and experiences.
I will disagree that science and scientists lack curiosity. This is, in fact, the primary intellectual driver of scientific research and discovery. Individuals, and certainly clinicians who are not researchers may get stuck in ruts and not look for new ways of doing things, but that is a problem with people, not with the method of science, which is constantly exploring new ideas. Unfortunately, many new ideas turn out to be wrong, and science frustrates people when it rejects new ideas even if that rejection is based on good evidence. People often mistake an unwillingness to believe claims with no better evidence than anecdote for closed-mindedness, but this is not the case. Being curious and open-minded doesn’t mean believing anything and everything.
Overall, I think we are in agreement, but I think individual human failings are inevitable and they are the reason science and skepticism is necessary, not an indictment of the scientific approach.
More people need to see this article. After my cat developed CKD I was scouring “Dr Google” for answers (hundreds and hundreds of tabs, I was so overwhelmed that I got a panic attack) as well as support groups online. SO. MANY. CONFLICTING. THINGS. The support groups were great overall (they helped me find a bunch of low-phos foods) but there were a few stragglers who suggested things with no scientific proof. There was even a woman who wanted to give her dying cat expired antibiotics … ??!? I found out the internet can be a useful tool, but always find a good veterinarian, work closely with them, and ask them before doing anything you read online, such as rubbing slippery Elm bark and colodial silver all over their inflamed gums -_- I believe pets are naturally very capable and durable and people fool THEMSELVES with placebo effects thinking whatever “natural” remedy they’ve given their pet made things better, when in fact, it could be masking things or making things worse. Take the “kidney support gold” shills for example, if you read the ACTUAL one star reviews, there are countless people who’s animals died horribly by taking it… All they wanted to do was help their pet and those $$$ hungry charlatans only made things worse. Less is more, in my opinion. If your vet prescribes something, use it and monitor for side effects, but don’t use any herbals without asking your vet first (again get a GOOD VET) and looking up the side effects and cautions (which there ALWAYS ARE) and unbiased negative experiences. I may have caused my cat’s CKD by working with a “holistic” vet and I’ll always hate myself for it. I may not have, too, though, but there’s no way of truly knowing. Now I’d rather my pet be affected by something mildly than die a few days after being given snake oil. If those alternative treatments really were so great, EVERYBODY would be using it, or cashing in on it… Think about it. Just my two cents.
Hum! Interesting. My veterinarian recommended me your blog. I read a good few numbers of your articles. Globally, I found them very interesting.
But then I came across three articles (including this one). It suddenly made me pause. I am neither for or against I would have said, traditional (but each clan could claim the word) medicine (human or veterinarian) so let’s say scientific medicine) nor am I for or against alternative (or whatever other word you might wish to use) medicine.
But it seems to me that you are as aggressive toward “non-conventional” practitioners that some of them are towards, let’s say, allopathic practitioners. Like in politics, it seems that in medicine for a large majority of people, it is either black or white. It is one or the other. At the end of the day, the patient (human or non-human species) is the loser.
My vet mentioned you when we were having a conversation regarding my dogs’ medications. I wanted to review them and their efficacies after I did online research on said medications via MSD Veterinary manual and other sources. I was mostly worried about side effects on prolonged usage. And I must admit I was quite surprised that some of those medications where not tested on dogs but on human and therefore being prescribed off-label. And sometimes, they are tested for other reasons that what they are used in Veterinary medicine.
According my vet, it is unknown if said medications are working or how they work if they do. But they have been used for a long time and are deemed safe. As my vet put it your oaths is do no harm and if a medication is deemed safe, it is OK to prescribe it. as it could do the animal some good. Nonetheless, those medications were not scientifically tested, neither for the uses they are prescribed for, nor on the species they are prescribed to.
Explain to me how that is not anymore quackery than practices that have been uses by humanity since immemorial times, even if science has not been able to prove yet if or how they work.
I am personally no more ready to use medications created by science that old women remedies. But I am enough open-minded that if I think that my immune system is not strong enough to fight by itself against diseases to look into external sources.
With my dogs getting older (13 and 18-year-old) and needing “medicines and treatments,” I am looking into everything. I might refuse to take anything for myself unless I am close to death or have reached my pain tolerance level. But I don’t want my dogs to suffer for my principles.
Now an other questions, what about pharmacovigilance? After all, it is based on testimonials and anecdotal evidences once a drug has been put on the market, after the end of its scientific trial. Should it be discarded? Since as you are insisting it cannot be trusted.
And what about the risks/benefits analysis needed before deciding to prescribe a medication?
Thank you for your time
To be clear, I am “against” making claims about any medical treatment that aren’t supported by reasonable scientific evidence. I don’t believe the category of “alternative” (or “holistic,” “natural” etc.) medicine really exists,. It is a marketing strategy to suggest that some things can be assumed to be safe and effective based on anecdote and opinion and don’t need to be validated scientifically. Any therapy we want to use should be evaluated by the same standards, from basic idea all the way through clinical trials, and the standard should be based on science. The fact that this works better than the thousands of years of trial-and-error and tradition and personal experience we were suing before the development of the scientific method is pretty clear in the dramatic improvement in the length and quality of human life over the last century.
The reason I have focused so much on alternative therapies is because the people who promote them are a lot less likely to feel the need for scientific validation than the rest of us, and these approaches typically have a lot less scientific evidence to support them than conventional, science-based therapies. That said, plenty of things considered “conventional” need to go because the science doesn’t support them, and I’ve made that case many, many times in writing and in educating vets. I just gave a talk last week to a veterinary conference on things we should probably stop doing, and almost none of them are labeled as “alternative.”
Off-0label use is an interesting problem. As you point out, without trials int he specific population we want to use a drug for, we have less evidence than we would like to show something is safe and effective. That said, veterinary medicine is the poor country cousin of human medicine, and medications tested and approved through the FDA process specifically for our patients make up a minority of the treatments we have because companies aren’t going to spend the time and money needed to generate the necessary evidence. So do we just give up most of our drugs? No. Instead, we try to strike a balance between the evidence we have (rather than the evidence we would like to have) and the need to do something.
In some cases, the need is not great (e.g. tear staining in little white dogs), and we shouldn’t use treatments without strong evidence. In other cases, the need is great (e.g. dogs with aggressive cancers), and a lower bar for the evidence may be acceptable. Heck, even very improbable alternative treatments can sometimes be reasonable to try, so long as we are clear with owners about the uncertainty and the risks.
There is no perfect balance between uncertainty and the need to act, but the closer we stay to a reasonable standard of evidence, the more likely we are to do more good than harm. Homeopathy will never help anybody’s pet, no matter how sick they are. Some things are just unreasonable. But there’s lots of grey area between that and drugs with robust clinical trial evidence supporting them.
If by pharmacovigilance you mean post-marketing surveillance, I disagree that this is purely anecdotal. The reports (anecdotes) are used properly to generate hypotheses (hey, lost of people are reporting Problem X with Dog A, maybe we should look into it). Theses reports can’t prove their is a problem, but they can suggest a subject we might want to study scientifically. That is always the appropriate use for anecdotes in medicine.