As we all know, the Internet is a mixed blessing. It makes enormous amounts of information easily and cheaply available to billions of people. Unfortunately, not all information is created equal, and sometimes bad information is worse than none. Contrary to what many of my clients assume, I actually appreciate it when they come to me with questions or suggestions based on their own research. However, when they have been misled by unreliable sources, a great deal of their enthusiasm and effort ends up being wasted.

I am hoping to contribute eventually to a more comprehensive set of evidence-based veterinary medicine resources under the auspices of the EBVMA, but in the meantime I have assembled some general guidelines and specific  sources to trust, as well as some to avoid. I hope this is helpful.

General Principles for Evaluating Internet Information

1. Consider the Source: We need EBM precisely because even the most intelligent and honest of us cannot avoid the influence of bias on our conclusions. The universal cognitive biases and perceptual biases we all share, as well as the more obvious errors associated with strong feelings and preconceptions make a truly objective evaluation of any complex phenomenon impossible. The scientific method compensates for such sources of bias imperfectly, but better than anything else yet discovered.

When evaluating information from the Internet, it is important to consider the possible sources of bias lurking behind it. These biases do not automatically invalidate the information by any means, but being aware of them helps inject a salutary element of skepticism about their validity.

Obviously, anyone trying to sell you something is likely to believe quite deeply in what they are selling. Apart from deliberate scam artists, most people don’t enthusiastically promote or make a living selling things they aren’t strongly committed to, so they are unlikely to be truly dispassionate or objective. Confirmation bias is particularly likely to lead people with a financial interest in something to ignore potential problems with it. CAM proponents never tire of making this charge against mainstream medicine, especially the pharmaceutical industry, but they conveniently ignore their own financial interest in the products and services they provide.

Strong ideological positions also call into question the level of fairness in the presentation of an argument. It is so difficult to overcome the influence of one’s deepest held values and philosophies that it is often most useful to simply declare them up front and let those who are trying to puzzle out the truth take them into account. When looking at information provided on the Internet, it is often useful to find sites committed to both sides of an issue and compare the information and arguments.

Sometimes such ideological biases may be hidden, usually in an attempt to prevent people from recognizing them and the influence they may have over the information being provided. However, one advantage to the Internet is that it is often fairly simple to find out a lot about the affiliations and associations of people offering information to the public. My recent post on the association between conservative Catholics and anti-vaccine activists is an example of how hidden agendas behind information presented as objective and scientific can be uncovered.

Finally, while blind reliance on “experts” is a form of faulty reasoning known as the argument from authority, it is still true that specialized training and experience in an area does give somewhat more weight to one’s opinions on the subject. The idea that anyone can be an expert on any subject simply doesn’t hold water in today’s complex and technologically sophisticated world. Personal experience is especially unreliable as a source of deep insights that an entire profession of specialists have missed. A few hours on the Internet can’t make me into a nuclear physicist, and it can’t make a physicist into a veterinarian. So while no one is automatically right just because they are a specialist or professional in a subject, it is unlikely that dilettantes and self-made experts are going to have insights or wisdom that such professionals lack.

2. Check References: In CAVM it is very common to cite scientific sources in support of mistaken arguments. It is also often the case that when one reads the original source, it doesn’t say what the person citing it claims it says. Taking information or quotations out of context, overlooking obvious flaws in methods or argument, and simply cherry-picking sources that say what you want them to while ignoring those that don’t are all practices that make such research citations unreliable. So whenever possible, check the original source to see if it really does support the argument you’re evaluating.

3. Be Reasonable: If it’s too good to be true, it probably isn’t. Extraordinary claims require extraordinary evidence. Pick your cliché, but ultimately reality is complicated and messy and often not how we’d like it to be, so anyone who tells you it is simple and you can have your cake and eat it too is probably deluded or lying.

Trustworthy Internet Resources

These are resources that I believe provide information that is reliable. Being run by human beings, they may not always be right, but they are not regularly, egregiously wrong or out to push a product. Some may have obvious biases in favor of science and methodological naturalism, but if one rejects these positions then one has already decided rational and objective inquiry is impossible or undesirable anyway, so trying to evaluate the rationality and objectivity of Internet information isn’t likely to be a concern. Postmodernists and those who see faith and revelation as the keys to truth aren’t likely to find any of this useful anyway.

Many of these resources address human medicine primarily, and many include subjects outside of medicine all together. There is an unfortunate paucity of good resources for science-based veterinary medical information on the Internet, though obviously I am trying to change that! however, despite the dangers of extrapolation across species, the information gathered in human medicine can be used to assist our judgments regarding veterinary medical questions.

The SkeptVet- Though this blog is the more active part of my Internet project, I have assembled a collection of my more detailed and researched articles and my downloadable client information pamphlets on specific veterinary topics on my main web site. No commercial conflicts of interest, and my ideological biases should be quite obvious.

Evidence-Based Veterinary Medical Association (EBVMA)– This is an organization of veterinarians and other professions, both in academia and private practice, dedicated to promoting evidence-based veterinary medicine. Any veterinarian interested in supporting high quality, science-based veterinary medicine should join and get involved. And a new web site, with practical information and EBM tools for vets is coming soon!

VeterinaryWatch– This site contains a lot of good, science-based information and references, though unfortunately not in a very user-friendly form. I am working with several others to try and improve the design and content of the site, but it is a slow project.

Quackwatch– By far the most extensive collection of resources regarding CAM. Reviews of many products and therapies, warnings about corporate and individual providers or CAM services and information, and links to many other reliable resources.

Science-Based Medicine– Far and away the best blog on science and evidence-based medicine and a great resource on CAM. Intelligent, thoughtful, and well researched essays on a large variety of topical as well as perennial issues.

The Cochrane Collaboration– The premier site for EBM in the human medical field. Though some resources are available only on a subscription basis, the most helpful resource are summaries of the independent and rigorous systematic literature reviews on a large number of specific topics.

PubMed– An enormous and easy to use database of the medical literature. Abstracts are available for most articles, and links to sources for the full text. Unfortunately, full text access for many articles is available only to subscribers of the publishing journal, but the world of medical publishing is changing, albeit slowly, and more journals become accessible all the time.

Free Medical Journals– A convenient way to identify journals that offer free online access to their content. While many of the highest impact journals in human medicine are open access, far fewer veterinary journals are available–yet.

Unreliable Internet Resources

While the number of commercial organizations and individuals pushing misleading information or outright nonsense on the internet is vast beyond imaging, this is a list of some of the more popular, and hence more dangerous. And while the distinction between bad science, pseudoscience, and plain quackery can sometimes be a tough call, a place to start for the more egregious cases is The Quackometer. This automated tool evaluates websites for language patterns typical of pseudoscience and medical quackery. Of course, I would never suggest letting a machine make your decisions for you, but I’m surprised by how often I agree with the little black duck!

Shirley’s Wellness Cafe– This is perhaps the most egregious woo site I’ve found to date. Vicious and paranoid condemnation of all scientific medicine as paternalistic, venal, and harmful. Wild claims about safe, natural, free cures for almost everything. No logic or data, only the presumption that if it is in any way scientific it must be bad for you. And despite the railing against the greedy medical establishment, there sure seem to be a lot of things for sale!

Healthy Pet Journal– A sad example of veterinarians giving up on science in favor of blind faith and intuition. The site doesn’t have the hysterical shrieking tone of Shirley’s, but that makes it even more dangerous since the information is presented as if it were reasonable and scientific, when it almost universally is neither.

Naturallycomplementary.com– A beautiful, well-organized, easy to use resource for avoiding science-based medicine and finding all things CAVM. A huge variety of unrelated and mutually contradictory approaches are promoted, with the only apparent unifying feature being claims of “natural,” “alternative,” “holistic” and so on. Words which have lost whatever meaning they may once have had and have become merely shibboleths for anti- and pseudoscience. I tried to register as a member of the forum to offer a different perspective, but apparently the management is not interested in allowing members to hear other points of view, and they denied my application.

Academy of Veterinary Homeopathy– A professional organization of veterinarians devoted to this tooth fairy science.* (sigh)

American Holistic Veterinary Medical Association– Ditto. This one is good for some laughs if you check out the offerings at their annual CE meeting.

Sadly, I could go on endlessly. While thankfully science-based medicine is far more widely available and accepted than CAVM, promoting CAVM on the Internet is a viable commercial activity, while promoting EBVM and skepticism is a strictly pro bono volunteer task. Consequently, the volume and shiny façade of the CAVM offerings will always outshout and outshine the more rational alternatives. But once you examine a few examples of unreliable sources, the patterns of language and argument become obvious, and your internal quackometer should steer you straight.

*”This study falls into the category of what I call Tooth Fairy science. You could measure how much money the Tooth Fairy leaves under the pillow, whether she leaves more cash for the first or last tooth, whether the payoff is greater if you leave the tooth in a plastic baggie versus wrapped in Kleenex. You can get all kinds of good data that is reproducible and statistically significant. Yes, you have learned something. But you haven’t learned what you think you’ve learned, because you haven’t bothered to establish whether the Tooth Fairy really exists.” Harriet Hall, MD

Yes, I’ll take the credit for seeing this one first. 🙂 The New York City Skeptics blog Gotham Skeptic has a post about recent pro-science articles appearing in the mainstream media. The post suggests we might finally be emerging from what a friend of mine refers to as the Golden Age of Woo and into a time when skepticism and sound science are again acceptable public positions to support.

They first mention Amy Wallace’s incisive piece in Wired magazine “An Epidemic of Fear: How Panicked Parents Skipping Shots Endangers Us All”, which has been discussed several times over at Science-Based Medicine. They then mention several other examples, including a piece by the Associated Press reporter Marilynn Marchione, whom I previously lauded back in June for her excellent reporting on CAM and pseudoscience. Hmm, I wonder if this might be evidence that I have psychic powers….. 😉

As I’ve pointed out before, the image of CAM providers as selfless promoters of health without the venal concerns for profit of mainstream medicine and Big Pharma is advertising spin, not reality. A nice example of that has been the recent attempts of many snake oil peddlers to cash in on fears about swine flu (H1N1 Influenza). The Food and Drug Administration (FDA) keeps a list of Fraudulent H1N1 Products. Of course, not all the products are CAM-related, since greed knows no loyalties. However, the majority of the products making unsubstantiated claims about preventing or treating the swine flu make similar claims about disease and health in general, and they include a number of CAM standbys.

Dr. Andrew Weil, a paragon of woo, has received a cease and desist order from the FDA regarding claims, subsequently removed, about his dietary supplement line. The FDA list contains a large number of herbs, vitamins, supplements, and even some teas. Despite the frequent claim that Big Pharma has no interest in these kinds of therapies because they cannot be patented and so there is no money to be made from them, there seem to be a lot of folks making money selling them for many uses, whether they work or not.

Some products are obvious attempts to cash in on flu fears, such as the Flu Away inhaler containing eucalyptus and tea tree oil. Others play more subtly on the fears of H1N1, marketing bogus “information” about the flu along with a broad collection of products and services designed to “promote health.” It is encouraging that the FDA is doing what it can, with limited resources and less public and even government support than it should have, to prevent unscrupulous individuals and companies from making a profit selling useless products and a false sense of security to people with legitimate concerns about the H1N1 pandemic.

Ok, after much prodding I have set up Facebook and Twitter accounts, so anyone interested in skepticism and veterinary medicine, or for that matter Celtic Folk music, science fiction, or an eclectic array or other inteelectual and aesthetic pursuits should look me up there! 🙂

Check out the 123rd Skeptics’ Circle. This is a regular blog carnival with posts from all over the world covering topics of interest to skeptics and critical thinkers. This week, you might even see a couple of posts you recognize! 🙂

When I was training to be a veterinarian, an older vet once described the nature of his work this way, “I make decisions for a living.” After more than 8 years in practice, I know exactly what he meant. The process of taking the information I get from a pet owner and from my physical examination and then using it to set up a diagnostic and treatment plan involves making many, many decisions. Which pieces of information are important and which irrelevant? How reliable are the owner’s observations? What information from the pet’s previous history might shed light on the current problem? What additional information do I need to make a diagnosis? What diagnostic tests are likely to be safe, useful, affordable, and logistically feasible? What is the true diagnosis and what treatment should I recommend? How do I assess the response to treatment, both beneficial and adverse effects? What do I do if the owner declines some or all of my recommendations? What ultimately is in the best interests of the pet and will best promote their health and well-being? And on, and on, and on for multiple patients all day every day.

This is often an intellectually exciting and stimulating process. It is also often frustrating, especially when information I need is unavailable, or when I make the wrong decision, as I inevitably will sometimes being human. One of the greatest challenges, in both a positive and negative sense, is dealing with uncertainty. Lay people often view clinical medicine as a rather mathematical process. The see the doctor as taking historical and physical exam information, and maybe some diagnostic test results, plugging them into an algorithm established by experience, tradition, and scientific research, and coming out with The Answer printed in black and white at the bottom of the paper like an old-fashioned adding machine. When the doctor expresses uncertainty or inserts caveats into his description of the diagnosis or treatment, the client is likely to assume this represents a gap in the clinicians knowledge or competence. The Answer is presumably out there waiting for the right doctor to ask the right question or run the right tests and find it.

While there are particles of truth in this image, it ignores many sources of uncertainty. I’ve already mentioned one which bedevils the veterinarian, which is the often surprising lack of knowledge owners may have about their own pets. Historical information is notoriously unreliable, as are the theories, often supported with internet citations, that owners bring as to the nature of their companion’s problem. Now I realize the owner is most often the single best source of information about the pet, but even the most astute owner may not notice things that only a trained health care provider would notice, and many owners unfortunately can’t even tell me whether their animal is eating or eliminating normally.

Obviously, the information that can be obtained directly from the patient is often limited. Animals can show signs of pain, pruritis, and other physical sensations that might indicate what the problem is, but they cannot give the sort of precise descriptions of their symptoms we might hope for. And despite the claims of innumerable self-professed dog/cat/horse whisperers out there, there are serious dangers to accepting as reliable one’s own intuitive impressions of what an animal is feeling, as I discussed in my post on Animal Hospice care.

Other sources of uncertainty include the fundamental limits of our understanding. Though the progress of the last century in understanding the complex relationships that constitute physiology and the various causes and risk factors associated with disease is absolutely astounding, and continuing at a breakneck pace, the reality is there is much we still do not know. And the volume of what is known is so vast that it is impossible for any one individual to have an adequate command of all the relevant and constantly changing information needed to properly diagnose and treat any condition. Atul Gawande illustrates this limitation in human medicine beautifully in his book Complications, and it is even more of a factor for veterinarians who rarely specialize to the extent MDs do and who treat multiple species.

And while I take a risk venturing into the territory of epistemology, I think it is likely that much of the true, underlying nature of complex phenomena, like organisms and their health or diseases, is probabilistic rather than strictly deterministic. Certainly, in medicine we can often say with great accuracy how a group of patients with a particular condition will do over time based on research, but we are rarely able to say with any certainty how an individual patient with the condition will fare. While this may be due, to a large extent, to the lack of adequate knowledge and measurement capacity, it may also be due to a fundamental unpredictability to complex systems. Chaos theory, quantum mechanics, and the philosophical approaches grouped under the label of  indeterminism all address this possibility, and while it is true that these theories may not always be applicable to the problems of  everyday life (especially in the case of quantum mechanics, which applies, as far as we can determine, only at the subatomic level), they do suggest that not all natural phenomena are going to be amenable to precise, accurate prediction regardless of how accurate and detailed our information about them is.

Given, then that we must make important decisions with significant consequences on the basis of imperfect information and with the understanding that we will never be as accurate in our assessments and predictions as we would wish, what is the best strategy for making the necessary decisions anyway? First, there is a psychological hurdle we must overcome. I have seen some new veterinarians absolutely paralyzed with indecision when they first encounter the imperfect world of clinical medicine outside of the university. And some eventually give up practicing because it is too difficult to accept the reality that they will make imperfect choices and mistakes despite their best efforts, and that some of these will result in harm to their patients. For most of us, though, we must find the confidence necessary to deal with this reality and to accept that we can only do the best we can and that however imperfect it is better than if we did not even try.

I have seen far more veterinarians, unfortunately, cope with the uncertainties of medical decision making through an excess of confidence in themselves and their choices. Given that we do not have the detailed accounting of outcomes that MDs often have in hospitals, there are rarely formal mechanisms in place for a vet to determine how well they are doing their job, either in an absolute sense or in comparison to their peers. Confirmation bias, cognitive dissonance, and many other normal human cognitive predispositions make it very difficult for us to accurately assess our own performance. Like most people, we will tend to see what we want to see, which is that we are competent, caring, and effectively helpful. And while this is true for most veterinarians, the reality is that we likely overestimate our own abilities and underrate our flaws just like everybody else.

Other vets will deal with uncertainty by slavishly following formulae. Memorizing simple guidelines along the lines of “If THIS, then THAT” and applying them to every case is comforting in that it essentially eliminates decision making all together. Of course, it also eliminates assiduous observation and critical thought, which are essential for the practice of good, sophisticated medicine. One of the common objections to evidence-based medicine is that it is simply a way to enforce this sort of robotic algorithmic approach on everyone. As I’ll discuss, this isn’t really true, but the objection is at least founded on an appropriate disdain for such an approach.

It will come as no shock to those who are familiar with this blog that my answer to dealing with the uncertainties of medical decision making is to apply the paradigm of science-based medicine (SBM — strictly speaking, this is different in subtle but important ways from evidence-based medicine [EBM], but for present purposes they are essentially the same). I view taking on the SBM approach as something a bit like embarking on a 12-step recovery program. Below, I outline and discuss some of the steps involved. (There is, of course, a certain whimsical tone in such a presentation, but the content is an honest representation of the approach I believe veterinary medicine needs).

1. We must begin by admitting we have a problem. This is probably the most important, and certainly the most difficult of the steps. We must acknowledge that our decisions are limited by inevitable uncertainties and that while we can never achieve perfection we can do better with help than we do with a traditional reliance on our own experience, intuition, knowledge, and skills. The greatest resistance I get from other veterinarians to SBM stems from the idea that they don’t need it because they can reliably assess themselves and their diagnostic and treatment approaches just fine. A lot of what this blog is about is pointing out how mistaken this excessive and misplaced confidence is. I acknowledge that it is a natural response to the distress caused by uncertainty, but it is not the right strategy to cope with this distress.

2. Having accepted our limitations, we must be willing to overcome them. This involves educating ourselves about the specific weakness and limitations that diminish the quality of our decision making, It also involves taking concrete steps to address them even though this means letting go of familiar patterns of behavior and strategies that we are comfortable with and trying out new and unfamiliar approaches. We must have an ongoing, never-ending willingness to identify and acknowledge weakness in our knowledge and our practices and the courage to make changes even in long and dearly-held beliefs and behaviors.

3. Accepting that we need to change, we must acknowledge that there is a higher source of knowledge and evaluation than experience or tradition. This is, of course, the scientific method. We will rarely have as high a quality of evidence as we might like given the financial and logistical barriers to clinical and basic veterinary medical research. But we must be committed to accepting the conclusions of the best available evidence as more reliable than those of lower-quality sources of evidence such as our personal experiences. If well-designed and conducted research contradicts my intuition or personal clinical experience, I must be willing to defer to the more reliable source of information.

I do understand how very difficult this is for people in general, and doctors in particular I suspect. But it has been shown time and time again that the beliefs and intuitions of human beings, even multiplied by many people over extended periods of time, are inferior to the conclusions of reliable scientific research. The progress in medicine and the increase in the length and quality of our lives since we began moving away from faith-based and tradition-based medicine and towards science-based medicine is dramatic and incontrovertible.

4. We must then proceed to educate ourselves, our colleagues, and our clients. We must set up systems for generating reliable scientific information and making it easy to access for general practitioners. The Evidence-Based Veterinary Medical Association (EBVMA) is a group dedicated to doing precisely this, and I encourage all veterinarians and pet owners to support their work. Setting up resources such as online journal access, evidence-based reviews along the lines of the Cochrane Reviews, and other such tools would make it easier for practitioners to employ a science-based strategy.

However, they will only turn to this model if we who understand its benefits can make the case to our colleagues that they should, for the sake of their patients and clients. By education and by example, we have to show other veterinarians how the inevitable uncertainties in medical decision making can be minimized by an SBM strategy.

And we must educate our clients so that they can be more effective participants in their pets’ healthcare. Clients who understand the nature of medical decision making will be better able to make sound, informed choices among the options offered to them by their veterinarians. Client values and resources are a key factor determining what we can do for our patients, so we must include them in the decision-making process for practical as well as ethical and philosophical reasons. And the frustration and miscommunication that all too often taints the veterinarian/client relationship will be lessened if our clients understand that medicine is neither a straightforward mathematical process with clear right and wrong answers nor a mystical and vague “art” that depends solely on the talent and wisdom of a particular clinician.

As I mentioned earlier, the objection often made to SBM and EBM approaches is that they seek to impose a cookie-cutter uniformity and robotic system of rigid decision trees on clinicians. This might be true in a deterministic world with the availability of perfect information, but in the real world there will always be uncertainties and gray areas that require a thinking and caring doctor. Contrary to the impression I sometimes give, I believe that clinical experience and intuition are forms off evidence. They are very low-level evidence, and far less reliable that higher-level forms, but they are often all we have to go on, and they can serve us well if carefully and judiciously applied, with great humility. And even when the evidence is strong and we can define the medical parameters of a situation well, medical decisions often depend on non-medical decisions about values and interests. We may be able to say very accurately what the likely treatment outcomes are for a particular disease given the alternative choices, but that doesn’t necessarily tell us what the right thing to do is for a given patient and their family.

And in order for the veterinarian and client to work effectively as a team for the best interests of the pet, the vet must be able to understand and communicate the nature of the medical situation, the options available, and the inherent uncertainties in the decision-making process to the pet owner. All of these are reasons why even in the best case where SBM guides our practice, there is a need for the experience and skills of the veterinarian.

Many of my biggest concerns about CAM and the philosophy that underlies much of it are related to how uncertainty is handled. Many of the inappropriate responses to uncertainty I see mainstream veterinarians take sometimes are proudly adopted as foundational principles in CAM. Often uncertainty is denied entirely, with CAM practitioners claiming all diseases can be definitively traced to a simple causal schema (unbalanced ch’i or humours, innate intelligence blocked by subluxations, toxins, and so on). Denying differences between individuals, species, or diseases and blaming all illness on universal vitalist forces gone awry is the ultimate abdication of responsibility for making careful, informed medical judgments.

Uncertainty in treatment outcomes are often denied as well. Worsening of the patient’s condition is frequently taken by homeopaths to be a sign of improvement, a so-called “healing crisis.” Of course, improvement in symptoms is also taken as a sign of improvement, so there’s no way to lose! As I’ve related before, CAM believers are often unwilling or unable to admit any room for improvement in the wisdom received from millennia or centuries or decades of tradition. While I believe the best response to uncertainty is open acknowledgement of it and an ongoing effort to improve and change, tradition-based practices are founded on the principle of preserving and not questioning received wisdom. And the reliance on anecdote and testimonial commonly seen in CAM venerates the individual experience and intuition. However, I believe less rather than more respect for this form of evidence is the key to more effectively managing the uncertainty in decision making.

Finally, in case I haven’t emphasized it sufficiently, I believe that a degree of uncertainty and unpredictability is intrinsic to medicine and we will never be free of it. Accepting this ourselves as veterinarians or our clients as owners to make perfect decisions is vital to providing good care. Recognizing our limitations not only helps us accept a better methodology, but it frees us from the illusion that we can be perfect or that we can control all outcomes for all of our patients. This makes us not only better doctors but, I suspect, happier people.