Early in 2010, the Practitioner Committee of the EBVMA conducted a survey of practicing veterinarians in the United States concerning their familiarity with the terms and concepts of evidence-based medicine and their attitudes towards it. The survey instrument was based on those used in published studies involving medical doctors and nurses. There were significant challenges in obtaining an adequately large, representative sample of U.S. practitioners. Ultimately, 5000 veterinarians were invited to participate via a printed letter, and 119 completed web-based questionnaires were completed, a response rate of about 2.5%. While this does not permit meaningful generalizations to be made about the population of interest, the project was, at the least, an instructive pilot study, and further studies are planned.

The only way to preserve the figures and formatting of the results summary was to attach it as a .pdf to this post, which can be viewed using the link below.

Survey of Veterinarians’ Knowledge and Attitudes Concerning Evidence-Based Medicine

One subject that I am perennially interested in is the nature of how people in general, and doctors in particular, make decisions and judgments, and how that process can go wrong. I’ve written about the pitfalls of spiraling empiricism, cognitive dissonance, uncertainty and medical decision-making, the Dunning-Kruger effect, why clinical experience is often unreliable, and other aspects of how general human psychology, and the attitudes, training, and approaches of doctors in particular, can lead us to erroneous conclusions and bad clinical decisions. Now a colleague has introduced me to the field of medical cognition, through a dense and often painful-to-read but fascinating and informative article:

Patel, VL. Arocha, JF. Kaufman, DR. A primer on aspects of cognition for medical informatics. J Am Med Inform Assoc. 2001;8:324–343.

There is a lot that I am not qualified to understand or evaluate in the article, particularly as pertains to the details of electronic medical record systems, artificial intelligence, and so on. But some of the insights gleaned from the cognitive psychology literature on how doctors develop decision-making strategies and how these change with education and experience seem very relevant to everyday clinical practice. One pretty established concept is that experienced doctors and those with highly developed skills and expertise are better able to filter out irrelevant information and attend to and classify what is most important in establishing a diagnosis. As the authors put it, research identifies:

…the greater ability of expert physicians to selectively attend to relevant information and narrow the set of diagnostic possibilities…The ability to abstract the underlying principles of a problem is considered one of the hallmarks of expertise, both in medical problem solving and in other domains.

One of the models for how experts achieve this is the concept of schemata:

…mental representations of typical things (e.g. diseases) and events (e.g. episodes of illness)…[which serve] as a “filter” for distinguishing relevant and irrelevant information. Schemas can be considered generic knowledge structures that contain slots for particular kinds of findings (data). For instance, a schema for myocardial infarction most likely contains the findings “chest pain,” “sweating,” but not the finding “goiter,” which is part of the schema for thyroid disease…A function of schemata is to provide a “filtering” mechanism to experts, allowing them to selectively attend to significant information and discard irrelevant clinical information.

Experts process information at a level of abstraction that is most efficient and reduces the burden on memory. Through years of experience, they have learned to conceptualize medical information (e.g., clinical findings from a patient) in terms of constructs…intermediate between the concrete level of particular signs and symptoms and the more abstract nature of diagnoses… In contrast, less experienced physicians tend to process medical information at a more detailed level.

Building these schemata and learning to process the huge amount of available information (history, physical examination findings, bloodwork, imaging, etc) efficiently and effectively takes a long time.

This research has shown that, on average, the achievement of expert levels of performance in any domain requires about ten years of full-time experience.

Developing such expertise certainly requires acquiring specific factual knowledge. However, research suggests that the importance of facts in building competence is often overestimated.

Factual knowledge involves merely knowing a fact or set of facts (e.g., risk factors for heart disease) without any in-depth understanding. Facts are current truth and may become rapidly out of date. The acquisition of factual knowledge alone is not likely to lead to any increase in understanding or behavioral change. The acquisition of conceptual knowledge involves the integration of new information with prior knowledge and necessitates a deeper level of understanding…Factual knowledge is inherently more brittle than conceptual knowledge, and this brittleness is most acutely observed in unfamiliar situations.

This certainly accords with my own experience of transitioning from a new graduate to an experience veterinarian. I sometimes feel as if much of the detailed factual knowledge acquired and regurgitated laboriously in veterinary school has left me, yet I am able to identify the important pieces of information in a given case and relate them to relevant criteria for diagnosis or treatment much more easily than new graduates I work with. And, of course, facts are always available to be looked up when needed.

Interestingly, research in the development of expertise does not seem to support the popular, conventional model of how one gets better at a complex skill. For one thing, the process does not seem to be a steady accretion of knowledge and skill, but an erratic, unsteady trajectory. And in many cases, as one shifts from a detailed, algorithm-driven, formalized method to the more efficient, heuristic approach of an expert, one’s competence may actually decline, a phenomenon the authors refer to as the “intermediate effect.”

Cross-sectional studies of experts, intermediates, and novices have shown that, on some tasks, people at intermediate levels of expertise may perform more poorly than those at lower levels of expertise, a phenomenon known as the “intermediate effect.” When novice–intermediate–expert data are plotted…the performance of intermediate subjects (those who are on their way to becoming proficient in a domain but have not reached the level of experts) declines to a level below that of novices…

This literature suggests that human development and learning does not necessarily consist of the gradually increasing accumulation of knowledge and skills. Rather, it is characterized by the arduous process of continually learning, re-learning, and exercising new knowledge, punctuated by periods of apparent decrease in mastery and declines in performance. Given the ubiquity of this phenomenon, we can argue that such decline may be a necessary part of learning.

One theory that occurs to me to explain this, and which the authors don’t appear to consider, is that the fundamental nature of the shift from novice to expert may itself be counterproductive in some ways. Novices tend to follow explicit rules and patterns taught to them for sorting and utilizing information and solving problems. Experts tend to have internalized these rules and often process information and draw conclusions without explicit, conscious awareness of the thought processes involved. While this is inarguably more efficient, it raises the risk of bias significantly. It appears to be well-established that the risk of drawing incorrect conclusions is increased when explicit and objective controls for unconscious bias are not utilized. Barry Beyerstein has created a list of the cognitive biases and errors that can lead to incorrect clinical decisions, and many of these would seem to involve relying on instinct or intuition, which are colloquial labels for exactly the kind of unconscious information processing the authors of this article characterize as the hallmark of an expert (I have adapted and modified this list to suit the veterinary profession).

Human Psychology Even when no objective improvement occurs, people with a strong psychological investment in the pet can convince themselves the treatment has helped. And doctors, who want very much to do the right thing for their patients and clients, have a vested interest in the outcome as well. A number of common cognitive phenomena can influence one’s impression of whether a treatment helped or hurt a patient. Here’s a brief list of common cognitive errors in medical diagnosis. Any of these sound familiar?

a.      Cognitive Dissonance When experiences contradict existing attitudes, feelings, or knowledge, mental distress is produced. People tend to alleviate this discord by reinterpreting (distorting) the offending information. If no relief occurs after committing time, money, and “face” to a course of treatment internal disharmony can result. Rather than admit to themselves or to others that their efforts have been a waste, many people find some redeeming value in the treatment.

b.     Confirmation Bias is another common reason for our impressions and memories to inaccurately represent reality. Practitioners and their clients are prone to misinterpret cues and remember things as they wish they had happened. They may be selective in what they recall, overestimating their apparent successes while ignoring, downplaying, or explaining away their failures. Or they may notice the signs consistent with their favored diagnosis and ignore or downplay aspects of the case inconsistent with this.

c.      Anchoring This is the tendency to perceptually lock onto salient features in the patient’s initial presentation too early in the diagnostic process, and failing to adjust this initial impression in the light of later information. This error may be severely compounded by the confirmation bias.

d.     Availability The disposition to judge things as being more likely, or frequently occurring, if they readily come to mind. Thus, recent experience with a disease may inflate the likelihood of its being diagnosed. Conversely, if a disease has not been seen for a long time (is less available), it may be underdiagnosed.

e.      Commission Bias results from the obligation toward beneficence, in that harm to the patient can only be prevented by active intervention. It is the tendency toward action rather than inaction. It is more likely in over-confident veterinarians. Commission bias is less common than omission bias.

f.      Omission Bias the tendency toward inaction and rooted in the principle of nonmaleficence. In hindsight, events that have occurred through the natural progression of a disease are more acceptable than those that may be attributed directly to the action of the veterinarian. The bias may be sustained by the reinforcement often associated with not doing anything, but it may prove disastrous.

g.     Diagnosis Momentum Once diagnostic labels are attached to patients they tend to become stickier and stickier. Through intermediaries (clients, techs, other vets) what might have started as a possibility gathers increasing momentum until it becomes definite, and all other possibilities are excluded.

h.     Feedback Sanction Making a diagnostic error may carry no immediate consequences, as considerable time may elapse before the error is discovered, if ever, or poor system feedback processes prevent important information on decisions getting back to the decision maker.

i.       Gambler’s Fallacy Attributed to gamblers, this fallacy is the belief that if a coin is tossed ten times and is heads each time, the 11th toss has a greater chance of being tails (even though a fair coin has no memory). An example would be a vet who sees a series of patients with dyspnea, diagnoses all of them with a CHF, and assumes the sequence will not continue. Thus, the pretest probability that a patient will have a particular diagnosis might be influenced by preceding but independent events.

j.       Posterior Probability Error  Occurs when a vet’s estimate for the likelihood of disease is unduly influenced by what has gone on before for a particular patient. It is the opposite of the gambler’s fallacy in that the doctor is gambling on the sequence continuing,

k.     Hindsight Bias Knowing the outcome may profoundly influence the perception of past events and prevent a realistic appraisal of what actually occurred. In the context of diagnostic error, it may compromise learning through either an underestimation (illusion of failure) or overestimation (illusion of control) of the decision maker’s abilities.

l.       Overconfidence Bias A universal tendency to believe we know more than we do. Overconfidence reflects a tendency to act on incomplete information, intuitions, or hunches. Too much faith is placed in opinion instead of carefully gathered evidence. The bias may be augmented by both anchoring and availability, and catastrophic outcomes may result when there is a prevailing commission bias.

m.   Premature Closure A powerful error accounting for a high proportion of missed diagnoses. It is the tendency to apply premature closure to the decision making process, accepting a diagnosis before it has been fully verified. The consequences of the bias are reflected in the maxim: ‘‘When the diagnosis is made, the thinking stops.’’

n.     Search Satisfying  Reflects the universal tendency to call off a search once something is found. Comorbidities, second foreign bodies, other fractures, and coingestants in poisoning may all be missed. Also, if the search yields nothing, diagnosticians should satisfy themselves that they have been looking in the right place.

It may be that there are advantages to the deliberate processes followed by novices, and that the adjustments in these made to achieve speed and efficiency aren’t always exclusively favorable to accuracy. Of course, there is evidence that experts truly are better at arriving at correct conclusions than novices, so the heuristic methods that they develop are effective most of the time. But a key element in encouraging the adoption of evidence-based medicine is inculcating adequate self-doubt. It is clear that explicit, objective methods of analysis are less prone to bias and error than reliance on our own perceptions and internalized and unconscious decision-making processes. This reliance on the explicit and the objective is most critical in controlled clinical research, but it is also a useful process for reducing error in day-to-day clinical practice. So in recognizing and emulating the heuristic practices of experts, we must not neglect to recognize their pitfalls and include tools and methods for compensating for these weaknesses.

This blog grew out of my own efforts to educate myself and my clients. I began looking into alternative medicine in veterinary school (about 14 years ago now, gasp!). Having heard about James Reston’s supposed acupuncture anesthesia in the 1970s, I assumed acupuncture was a generally effective therapy, and I wanted to learn how to use it. However, the more I learned, the less I believed about the claims made for acupuncture, and the more aware I became of the role of our beliefs and desires in interpreting the evidence of our senses. I gradually learned to have a healthy awareness of our inherent limitations as human beings that lead us to mistaken beliefs about health and disease, and of the great value of science, however imperfect, in compensating for these limitations.  

Once in practice, I was called upon almost daily to offer my professional opinion on many common questions from clients about how to best restore or maintain their pet’s health. In response to these questions, I began actively and critically investigating and evaluating the evidence for a wide range of medical interventions, both conventional and alternative. I began writing a series of handouts on these questions, covering common CAM therapies such as homeopathy, acupuncture, chiropractic and so on, as well as questions about conventional interventions such as neutering and vaccines. This series, and my growing personal and professional involvement with evidence-based medicine, led to this blog.

The primary purpose of this blog is to make both the specific information I discover and the general principles of science-based medicine available to anyone who is interested. While SkeptVet is a nearly microscopic fish in the Internet ocean, readership has grown quite a bit over the nearly two years of its existence. As near as I can tell from the comments and emails I receive, my audience is a mix of veterinarians and other veterinary professionals and pet owners and breeders. I sometimes get great questions which lead me to investigating and learning about subjects I would have otherwise been unaware of. And I get some very rewarding feedback, by comment or email and, occasionally in person, from people who have found my posts informative, interesting, or helpful in some way. That keeps me slogging away since that is, after all, the point.

But of course many of the responses I get are highly critical. Some of the criticism I receive is thoughtful, informed, and civil and provides a beneficial and necessary impetus for me to challenge and question my own beliefs and assumptions. That I always welcome. Unfortunately, most of the critical responses I get are not thoughtful, informed, or even civil. There are a few “stock” ideas in many of these responses, and quite a variety in terms of styles and degree of eloquence.

My general policy is that I approve any comment to the site regardless of the content so long as it is not simply gratuitous personal abuse or completely unintelligible (and, of course, spam which is blocked automatically for the most part). I have only had to block about half a dozen comments since the blog began, so I think my policy is quite liberal. However, it amazes me how irate some people get at the mere fact that I moderate comments at all, as if I were somehow obligate to let them post anything they like, no matter how vitriolic or incoherent, on a site I created and maintain as an informational resource.

I thought it my by interesting, and even entertaining, to take a look at some of the criticism I receive and to see if any meaningful patterns can be. So far, I’ve come up with three basic categories (and excluding, of course, the most vicious and profane, which I simply delete unless I think it may one day be needed to obtain a restraining order).

I. Who the hell do you think you are? I’m as smart as you and I’ve seen XX work, so who cares what you think? Science doesn’t know everything and neither do you!

I’ve tried to address the issue of anonymity and my background, as well as the issue of how unreliable the “I’ve seen it work” argument is, in the FAQ, as well as in many posts and comments I make. Still, this is by far the most common critical comment I get. Here are some examples.

My dog has been diagnosed with Cushing’s disease 2 years ago. She was very sick. I did what the vet said but it all just got too expensive. I tried Superglan and she doesn’t even seem like she has the diesease. I think everyone has their own opinion and if you want to try it, then try it. Who cares what everyone else thinks!

Traditional Chinese Medicine has been around for thousands of years, and is a proven methodology of health care. Just because it doesn’t follow the Western medical culture doesn’t mean it doesn’t exist!… It is a shame that no Western medical studies have been done on this product- because that is the only way that close-minded people like you will ever even give it a try.

I started off taking you seriously, but after reading a few of these exchanges, you lost all credibility with me. Someone who was truly unbiased would at least consider the fact that it MIGHT work. Even if it has not bee scientifically proven, a true skeptic does’t come to ANY conclusions without evidence. You have clearly come to a conclusion without evidence, and your bias became clearer and clearer the further I read. Bias is nearly always based on something personal. So did you have a negative experience with this product, or a similar one, or are you being paid?… basically this was just a rant attacking a lack of controlled study, and showing your disdain for intuition and the powers of observation of pet owners. If you want to discard human powers of discernment and behave like a robot, or a computer, that’s fine…Clearly you expect to only encounter only non thinking persons. Your whole act of “Let me, the expert, the person who is more informed than you, enlighten you about critical thinking and scientific study so you do not harm yourself in your ignorance” is just as bad as any snake oil salesman. You can only peddle that crap to someone who doesn’t know enough to spot bullshit and call it…If this is just a hobby, playing expert on a product based just on looking at a website… well… Maybe you should try going out? Or knitting? Something productive?

My arguments are based on practice and outcome. I fix dogs that have suffered from commercial dog food related diseases with plenty of cases in hand…It’s not about science, it’s about results that speak for themselves.

You lack humility and exude arrogance not because you follow your conscience but because you believe that those who don’t agree with your premise must be wrong. You say that, “Being critical of bad ideas and false claims is also not arrogance, it is conscience.” How is it that you have the wisdom to decide which ideas are false and which claims are bad?… When confronted with evidence from pet owners and other vets you dismiss it with the back of your hand as “unscientific”. That is the hallmark of arrogance from a pseudo intellectual. You essentially are saying that your opinion is better than their observations. That is what I meant when I suggested that you need to develop a little humility….You ought to consider using that superior intellect of yours to actually do some research and solve some medical mysteries. To paraphrase Woody Allen, those that can’t do teach and those that can’t teach author critical blogs about those that do.

Had I found your website first, I likely would have not ever tried Supraglans. Im glad things worked out the way they did. As far as testimonials, or more specifically, personal experience with this product being worthless and not scientific enough to be valid, I thought clinical trials were pretty much a give it to em and see if it works deal.

It seems like I exposed your thin skin and tweaked one of your ivy league nerves. Maybe it’s the old doctor-as-god complex rearing its ugly head again. How dare a mere civilian untrained in the ways of the medical profession question my superior authority and intellect when I have the scientific method to validate my truth. Once again, even with the scientific methods that you are so proud of, the “truth” evades you…Your post, more than anything I could ever write, shows the incredible arrogance of your profession. Do you not see it? Are you so full of yourself that you can’t open your mind to the possibility that you don’t possess all of the knowledge worth knowing in the healthcare universe?… You, even with your so-called scientific methods and Ivy League education are no better than me or countless other caring pet owners who don’t care where you went to school or how many footnoted treatises you read to formulate your scientific opinion. You are wrong as much as you are right that’s why they call it “practicing medicine”. Keep practicing, good doctor; maybe someday you will get it right.

If you have had extensive experience in an alternative field… but months, if not years in an alternative field, and you still found that it did not work…. I would commend you for having an open mind and actually trying it. But you seem to be putting down every modality without experiencing it for yourself. Have you tried every modality to validate everything you write about? You can’t write about acupuncture and say it doesn’t work, if you have not followed a case from begining to end, with an experienced vet doing it… and found that it did not work….I think your problem is not that there is no evidence saying that alternative medicine works, but the fact that you have your fingers in your ears like a little kid saying lalala I can’t hear you. When you live your life with blinders on, you miss out on alot.

With nearly half of all dogs passing from cancer, the cancer rate in cats nearly doubling since the distemper vaccine was made a yearly thing, I think we need to look beyond the blinders that were placed on our faces. We don’t have to be one extreme or the other. We need to all come together for the health of our animals, and our own health. ..This type of aggression, stress and hate is extremely unhealthy for us. High blood pressure and heart attacks to name a few.

i don’t care what research has been done the evidence is in the results… But when you use it on a daily basis and see results you will never be able to tell me it doesn’t work because frankly it’s just not true.

All I have to add to this is that acupuncture does work. I’ve seen it work. To me that is all that matters.

II. You are only saying this because you’re afraid you’ll lose business. You must be a tool of Big Pharma. You vets make more money if our pets are sick, so you don’t want us to have safe, cheap, effective natural remedies.

Obviously this website is biased against complementary integrative veterinary medicine….keeping comments and ideas one sided and supported pharmaceutical and commercial pet food monopolies which have been raking in the money for many decades…Threatened financially and ideologically, they must resort to political tactics of attack, shock and awe using headlines inspired by the National Inquirer or some other ladies gossip rag…If you were a legitimate blog looking for the Truth and not a shill for the pharmaceutical companies, you would have researched both sides of any issue.

The old 70 year old urologist told me I was crazy for believing that acupuncture helped my stone. I think he is crazy for thinking I would have let him make another 15 grand off of a surgery that caused me more pain and suffering than I had ever experienced. I wish someone would have told me that I could have just bought a 12 dollar bag of herbs. They don’t taste so great, but hey, it beats the heck out of feeling like the mob got a hold of your kidney with a bat. Or should I say the rich Mob Doctor MD.

I believe that traditional veterinarians are today motivated by GREED and the medicine they practice does as much harm as much as it helps…And how about the local vet’s push for more and more, now found to be harmful, vaccines they are always telling us pet owner are needed- just so they can make more money, not to mention the those oh so toxic flea meds,

it looks like the only faith you have is in your holier than though self. You remind me of our consulting vet. you can show him and show him, but when it comes down to the bottom line what the drug companies will do for him, he will jump on their bandwagon even when it doesn’t work

After reading your “rant” (so accurately described by another reader), I am left feeling like you must have a financial interest in big pharma, for that is the only logical reason I can see that you would put such effort in condemning a product that has successfully treated and prolonged the lives of so many animals… Maybe you should conduct a clinical research study with neo and see the results for yourself? Oh wait, you wouldn’t make the money using neo as you do with chemo and radiation, my bad. It sure does seem like your motives are financial,

III. Just plain angry

Thanks to scientists. we don’t believe in religion any more. Huge department stores have become our Mecca. Now we are told CAM is bunk and not to believe it.

What I object to is this dictatorial attitude of medics that we should obey them or else, and their efforts to shut down anyone who opposes them. They have become the new Popes of the scientific religion!

Its a shame there are Veterinarians out there that think like you and thankfully they are becoming fewer by the day…It would be a sad day for all animals if they had only you to treat them for cancer.

All this talk about science vs. marketing and that only a vet (or doctor) that practices with prescriptions and surgery are worth anything. This is the same mentality that has almost all of Americans frail and sick! My god, when is every person going to realize that we are living breathing cells who need nutrition and balance, not knives and drugs