One subject I am especially interested in is the ways our inherent reasoning processes can lead us to develop and defend mistaken beliefs. Many features of human observation and thought have evolved for quick judgments and efficient approximations under conditions of limited information. The disadvantages of these heuristics is that they are more susceptible to bias and some kinds of error than careful, formal reasoning processes. As veterinarians, the conclusions we make about our patients and our own treatments are subject to all the same sources of error that bedevil all human thinking.
I have written before about cognitive bias and clinical decision making in veterinary medicine. Now a new series of articles in the Journal of Feline Medicine and Surgery has taken on the subject of clinical reasoning and cognitive bias. This entertaining and informative series should be a valuable resource for veterinarians but also for pet owners interested in how decisions are made in the care of their pet. Some of the articles are freely available, though not all are yet.
Had a great time chatting about alternative veterinary medicine on the Prism Podcast. You can download the mp3 below if you like or listen on the web or on iTunes.
After nearly 100 hours on online instruction and independent study, my acupuncture training has concluded with a 5-day hands-on practicum and both a written and practical exam. All that remains to do to finish my certification in Veterinary Medical Acupuncture is a brief case report. This seems, then, like a good opportunity to wrap up SkeptVet’s Acupuncture Adventure and see what, if anything, has changed in my views and understanding of acupuncture as a result of this process.
Clinical Intensive Portion of the Acupuncture Course I will begin by sharing my experience in the practical training portion of the course. Many folks who know my skeptical and science-based views have asked if I met a lot of strange people or dedicated adherents to pseudoscience in this course, but that really wasn’t the case. For one thing, this particular course is heavily marketed as science-based and an alternative to the TCVM approach to acupuncture, so it tends to attract a more mainstream set of students. And it is important to remember that despite the significant limitations and problems in the acupuncture literature, it is pretty widely accepted in the medical community as having at least a few legitimate benefits. While I’m still not completely convinced this is true when one looks into the literature deeply and critically, I recognize that many vets and MDs who are genuinely science-based in their views and practices view acupuncture as perfectly consistent with that approach when it doesn’t involve too much talk of Qi and other TCM metaphors.
I also think we have to be wary of the natural human tendency, as strong among skeptics as those who believe in alternative medicine, to stereotype and caricature people who disagree with us. While there are interesting trends and correlations to investigate between temperament and a variety of views and beliefs, the world is not neatly and rigidly divided into the skeptical and the credulous. I have always emphasized that even believers in the most egregious nonsense, such as homeopathy, often turn out to be as intelligent and reasonable on a day-to-day level as the most committed skeptic. Perhaps the greatest challenge to skepticism and rationalism is the fact that the road to irrational or pseudoscientific beliefs is paved with many normal, ubiquitous humans cognitive features we all share. It would be easier to mitigate the dangers of pseudoscience if only the stupid or the gullible believed in it, but that isn’t reality.
As a skeptic, I am routinely assumed to be negative, argumentative, cynical, closed-minded, and generally unpleasant by people who know nothing of me other than my views on alternative medicine. This is unfair and inaccurate, but it is no more unfair or inaccurate than assuming anyone who might be interested in learning acupuncture must be gullible or some kind of caricature of a New Age hippy.
Generalizations are sometimes useful and interesting, but they are also a hair’s breadth away from the kind of stereotyping that impedes accurate understanding. For those of you who haven’t seen it yet, here is Tim Minchin offer some thoughts that strike me as relevant to this issue.
For the most part, the students and instructors I talked to in this course seemed drawn to acupuncture for fairly typical reasons. Some simply felt frustrated by the lack of validated therapies for many important clinical problems and felt that exploring something new might be worthwhile. Given the poor quality research evidence for much of conventional veterinary practice, the lack of strong scientific evidence for acupuncture didn’t necessarily seem to be a deterrent for most.
And of course, I heard a lot of stories about personal experiences that had drawn participants to acupuncture. As always, in conventional medicine nearly as much as alternative medicine, anecdote is king. Some students had experienced improvement after receiving acupuncture treatment themselves for medical problems that had been persistent or otherwise recalcitrant to treatment. Others had seen patient who were very ill, had poorly understood conditions, or had been expected not to recover improve dramatically after acupuncture treatment. Almost everybody had some experience with acupuncture, laser, massage, or some other kindred practice that they felt was of benefit to them.
These experiences naturally create a very compelling sense that the improvement must be due to the therapy, despite all the reasons why this may not be true, and the long history of the failure of anecdote-based medicine. And there is an equally natural and unfortunate tendency for one such experience to lead people to be more accepting in general of anecdotes as evidence, even for the more implausible treatments. Many prominent advocates of alternative practices describe such “conversion” experiences as the start of their CAM careers. There is sometimes a sense that if one claim for an alternative therapy turns out to be true (or at least seems to), then they must all be taken seriously. Obviously, this is fallacious reasoning, and each claim and modality has to be evaluated independently regardless of whether it carries some vague and often misleading label as “integrative” or “Western” or something else.
Unfortunately, very few vets have the time, training, or inclination to critically appraise the scientific literature when investigating therapies that are new to them. And sadly, the limitations in the veterinary literature (and the scientific literature generally) make much of it less reliable a guide to what works and what doesn’t than we might wish. This only strengthens our reliance on personal experience and anecdote.
This is, again, a natural and universal aspect of human reasoning. But while anecdotes provide both a wonderful source of testable hypotheses and at least a somewhat rational basis for specific practices pending the development of better evidence, they are, of course, a deeply unreliable basis for medical practice.
Much of veterinary medicine, again conventional as well as alternative, is largely anecdote based, and we have to live with this in the absence of better quality evidence. But this requires a solid understanding of why we should have only very weak confidence in practices that have not been scientifically validated, regardless of our own apparently miraculous personal and clinical experiences. Unfortunately, we do a very poor job of providing veterinarians with this understanding during their training. The students and instructors in this course were, in general, no less critical in their thinking and no more susceptible to the allure of anecdote and personal experience than those I meet in purely conventional practice.
The practical training itself struck me as a mixed bag. There was a strong emphasis on familiarity with anatomy, which is always a worthwhile way of approaching the physical exam. I found this a useful refresher, given I am 15 years out of school and haven’t formally studied anatomy in a long while. And I felt like the folks doing physical therapy and rehabilitation had a lot of useful insights into evaluating patients with musculoskeletal problems. While there is little data on physical therapy interventions in veterinary medicine, this is a well-established and useful dimension of human medicine, and it is highly likely it will be a useful addition to companion animal medicine, though as always the research identifying which practices are useful for which conditions and all the other important details will lag well behind clinical implementation of these practices.
There was a heavy emphasis on myofascial trigger points as a focus of diagnosis and treatment. I have discussed this topic before. The subjective nature of this phenomenon was very apparent, as different students and instructors often disagreed about the location and significance of MTrPs they felt they detected. In some patients, areas of muscle tension with repeatable behavioral responses to palpation did seem pretty obvious. But in other cases, confident claims about MTrP detection were made based on pretty weak behavioral responses and with great inconsistency in the opinions of different individuals examining the same patient.
As an example, one instructor was adamant about a MTrP being present at a specific spot on one dog. I could not appreciate this myself on palpation. As I was palpating, she noted the dog flick its ear and declared that as proof I had found the point. However, she couldn’t see from her position that the ear twitch occurred when I had accidently brushed that ear with my other arm, and I suspect the twitch was just a reaction to that mild direct irritation of the ear. This doesn’t mean there wasn’t a MTrP that the instructor could feel and I couldn’t, of course. But it does illustrate the problem of subjectivity and confirmation bias in MTrP theory.
While I am still agnostic about the MTrP as a clinical entity, I am concerned that with the strong element of subjectivity in its detection it seems a less than totally reliable concept upon which to base physical medicine interventions. That said, once again such subjectivity is ubiquitous in veterinary medicine, and the detection of pain, nausea, anxiety and many other clinically important variables are no less subjective than the detection of MTrP, so that alone isn’t sufficient reason to reject the concept.
The portions of the training specifically devoted to needling were interesting and also a mixed bag. The research evidence used to justify needling was mostly covered in the online portions of the course, so there wasn’t much discussion of that in the practical training. Some instructors were careful to make only relatively measured and limited claims for acupuncture, acknowledging that the evidence is complex, limited in significant ways, and generally not definitive. On the other hand, almost everyone had a quite obvious personal conviction that acupuncture was highly effective based on their clinical experiences. As I’ve already discussed, this is a ubiquitous and natural, but also deeply problematic, element of both alternative and conventional veterinary medicine.
I do think this type of course, ostensibly treating acupuncture the same as any other therapy in terms of striving for a plausible physiologic rationale and recognizing the need for pre-clinical and clinical research validation, is less faith-based than the sort of course provided by the Chi Institute and other TCVM organizations. There is at least some recognition that much of the practices being taught are not supported by high-level or high-quality evidence and that this is less than ideal. However, anecdote and personal experience remain sufficient in the minds of most of the practitioners, and what evidence there is tends to be interpreted in the most positive possible way. Negative studies are consistently dismissed due to methodological limitations, while positive studies are much more readily accepted as valid, and the general paucity of good evidence in veterinary medicine is seen as tacitly justifying the use of acupuncture despite the weakness of the science behind it.
On a practical level, I did find the application of needling to the many animals used in the practical training somewhat reassuring. Virtually none of the animals manifested any obvious anxiety or discomfort about their treatment. A variety of distractions were used, mostly food related, however many of the animals (mostly owned by the instructors or their clients) were quite comfortable and relaxed during treatment, even by inexperienced students, with no need for distraction at all.
In general, the needles used are extremely small (0.1-0.25mm diameter), and certainly far smaller than those I use routinely for blood and urine sampling, aspiration of masses and internal organs, and many other routine conventional procedures. I had a number of opportunities to needle myself or be needled by others, and while I didn’t experience any of the physical or emotional sensations often described as evidence of the powerful effects of acupuncture, I also didn’t find it particularly uncomfortable. Even the electrical nerve and muscle stimulation I experienced was pretty benign at the usually recommended levels of intensity.
Given that the evidence for benefits from acupuncture is poor, I think it is important that we not expose patients to significant discomfort or risk if we are going to try it for conditions that have no validated treatment. The instructors were all very cautious about causing any discomfort and very clear about the need to avoid reported complications, such as puncturing body cavities or vital structures. Such complications are reported for acupuncture, and some of the instructors had personal experience with them. However, even reviews of the literature by skeptics show that such complications are rare and usually associated with failure to follow standard safety recommendations or the result of needling done by improperly trained practitioners. So while the training didn’t add much in terms of evidence for the efficacy of acupuncture, it was reassuring in terms of reaffirming that the risk and discomfort associated with the practice are minimal.
View on Acupuncture from Start to Finish At the beginning of this course, I summarized my starting views of acupuncture. In brief, my conclusions were these:
My current view is that while needling undoubtedly has physiological effects, it has not been convincingly demonstrated that these are predictable, repeatable, and controllable to achieve beneficial clinical outcomes. I also am not convinced that acupuncture points or channels exist as a consistent network of identifiable anatomical structures that can be predictably identified and manipulated to achieve a desired clinical goal.
Ultimately, much of the acupuncture literature cannot be viewed as very reliable due…[various] sources of bias. The best controlled studies seem to suggest that acupuncture affects subjective symptoms and perceptions more than objectively measurable indicators of disease. This is most consistent with a placebo effect, and perhaps some very non-specific physiologic effects. However, there is some room for rational uncertainty about the extent to which acupuncture might have small benefits in terms of pain, nausea, and a couple of other clinical symptoms.
There are definitely risks associated with acupuncture…Infections and injuries from needles, sometimes serious, have been reported. Overall, serious adverse effects seem to be quite rare when experienced, formally trained acupuncturists are doing the needling.
Not much in these conclusions has changed as a result of taking this course. I have had an opportunity to review the experimental literature at length. There are a few studies, almost all involving electrical stimulation of peripheral nerves, that are pretty convincing in terms of showing objective physiologic effects of needling that might have clinical relevance. However, as yet there are few veterinary studies that control for bias well enough to allow any firm conclusion about whether these potential benefits can actually be consistently realized in clinical patients.
David Gorski has described electroacupuncture as a “bait-and-switch” with electrical nerve stimulation as an example of the “bait,” a potentially real intervention labeled in such a way as to appear to validate a much larger, and less plausible collection of concepts and claims. I think there is some legitimacy to this argument. If some traditional acupuncture points fall close to peripheral nerves or other anatomic structures we might reasonably want to use as targets for medical treatment, this doesn’t show that the overall pattern of channels and points used, even if acupuncturists could agree on only one, is a rational or scientific schema. And even if electrical stimulation of specific nerves using needles, or potentially needling in some locations without electrical stimulation, turns out to have consistent and objective clinical effects, this doesn’t make any and all claims made for acupuncture plausible or sound any more than the efficacy of one antibiotic for one type of infection validates the use of all antibiotics or even all drugs. Specific interventions need to be evaluated individually.
That said, I also think that if there is some “bait” in acupuncture, some objectively measurable effects of needling with or without electrical stimulation, it is worthwhile taking this seriously even if it is “tainted” to some degree by association with TCM or other pseudoscientific acupuncture practices. If one throws out the mystical energy medicine nonsense, pulse and tongue diagnosis, and all the religious and philosophical trappings of TCM, there remains a degree of plausibility to the idea that some needling and electrical stimulation interventions, labelled as “acupuncture” or by some other name, may be useful. The evidence for this in veterinary medicine, at all levels from pre-clinical to clinical trial, is poor but no worse than for a surprising number of conventional interventions.
Skepticism requires a careful, critical, nuanced evaluation of claims, relying on scientific evidence where possible and proportioning confidence in our conclusions to the strength of the evidence. At the beginning of this course, I had a pretty low level of confidence that acupuncture has meaningful clinical benefits, and that is still true. But I began with the belief that “there is some room for rational uncertainty about the extent to which acupuncture might have small benefits in terms of pain, nausea, and a couple of other clinical symptoms,” and that remains true as well. A detailed review of the evidence presented in the course leads me to believe that acupuncture is generally safe, probably objectively neutral with some placebo and other mild non-specific effects in most cases, and potentially truly beneficial in a few applications, especially those involving subcutaneous electrical stimulation of peripheral nerves.
What’s Next? So what does this lead to in terms of my clinical practice? I have mentioned before that I am considering offering some kinds of needling or electrical stimulation practices under the rubric of “acupuncture.” My reasons for this include: 1) the small, but not non-existent, potential for clinical benefits; 2) the lack of better-validated treatments for many important clinical problems where pre-clinical research suggests needling might be useful; 3) the low risk; and 4) the desire to avoid seeing clients fall into the hands of TCVM vets or other practitioners they are currently being referred to for acupuncture and who consistently undermine their use of science-based therapies.
Some readers have expressed concern about my potentially offering acupuncture treatments to patients. This concern appears to be founded in a number of particular objections.
Some have a strong confidence that no version of what is often called “acupuncture” can possibly have any clinical benefits, so any use of it would be unjustified. I have already addressed this in my detailed analysis of the evidence concerning specific acupuncture practices. While my confidence in the potential benefits of most specific needling practices is low, it is not zero for all of them, as it appears to be for these readers.
It is, of course, a philosophical issue whether or not it is ethical to offer treatments with only weak supporting evidence, and under what circumstances such might be legitimate to try. My own feeling is that when a practice has at least some evidence of plausibility (it could work), and perhaps even some weak evidence of clinical effects (it might work), even if it doesn’t have strong supporting clinical trial evidence to show it does work, it can be acceptable to offer such a therapy. One condition for doing so, in my opinion, is that any claims made must be proportional to the evidence, so clients need to be told that the hoped-for benefits are speculative. And if the benefits are uncertain, the risk and discomfort associated with the practice must be minimal, and it must not be used as a substitute for any better-supported treatment. I do think there are a few circumstances in which acupuncture can be justified under these conditions.
Other readers have suggested that needling animals is “torture” due to the discomfort it causes in patients who can’t give their own consent for the experience. Again, while I believe we must tolerate very little in the way of risk and discomfort when our confidence in the benefits of our therapies is low, my belief is that acupuncture is very unlikely to cause injury or pain, certainly less so than any other conventional practices I use routinely and which also have limited and weak evidence of benefit.
My own biggest concern about potentially offering acupuncture is the problem with distinguishing the many and incompatible associations of the term. Given it is pretty safe, needling of muscles and electrical stimulation of peripheral nerves strike me as reasonable interventions when used for specific indications that have some supporting evidence (e.g. pain, vomiting, GI motility disorders, etc.), when there are no better validated interventions available, and when owners are clearly and honestly informed about the lack of strong evidence. However, just as suggesting this may strike some skeptics as evidence that I have “drunk the Kool-Aid” that is TCVM acupuncture, so it may strike some pet owners as an endorsement for all the varied claims and practices labeled “acupuncture,” including those which I think are utter nonsense.
Distinctions between acupuncture as a form of “percutaneous electrical nerve stimulation” versus a means of “tonifying Yang” and “moving Wind” are meaningful but likely to be lost on many lay people. I share the concerns others have raised that even if I stick with the few plausible needling practices and am very careful about limiting my claims and explaining my rationale, I could still be misinterpreted as supporting a range of claims and myths associated with acupuncture which I actually see as pseudoscience.
A popular metaphor for “integrative medicine”, which I have quoted many times, comes from Mark Crislip. He has said, “If you integrate fantasy with reality, you do not instantiate reality. If you mix cow pie with apple pie, it does not make the cow pie taste better; it makes the apple pie worse.” In the course of digging through the claims and evidence concerning acupuncture, I have come up with a related simile that fits my feelings about this topic a bit better.
I believe that the few potentially beneficial practices in the acupuncture collection are like ponies buried under a large pile of horse manure. The trick is to dig the ponies out and put them to work without getting thoroughly covered in manure. I’m not sure exactly how to go about this, but I also feel obliged to not ignore even the limited potential for benefit to patients in those cases where there is some reason to think needling and electrical stimulation might be helpful and better-validated options have failed or don’t exist.
Since I think these are interesting and important issues, for veterinarians and pet owners, I will continue to share my thoughts and experiences about them here. I welcome questions and comments from both “sides” of the issue, though as always I prefer substantive and civil input. Thanks for coming along on SkeptVet’s Acupuncture Adventure!
I’ve written in the past about the subject of cranberry extracts used to prevent or treat urinary tract infections. (1, 2) There is some in vitro and lab animal research that suggests these products might reduce the ability of a limited set of bacteria (fimbriated E. coli) to stick to the lining of the urinary tract, and might thus have some value in preventing, though not treating, urinary tract infections. However, the latest Cochrane systematic review on the subject did not find convincing evidence these products actually are effective in UTI prevention in human patients. There is little research in veterinary patients, and unfortunately a new study in the American journal of Veterinary Research (AJVR) doesn’t add much information to help in deciding whether or not such products are useful.
This study consisted of two parts. The first involved treating dogs with a history of frequent urinary tract infections with either an antibiotic or a cranberry supplement and monitoring them for development of UTIs. The other involved mixing E. coli and canine kidney cells with urine from dogs given the cranberry supplement to see if there was any antibacterial effect or any impact on the adhesion of the bacteria to the kidney cells.
In the first part of the study, 12 dogs were split into two groups (it’s not clear if this was done randomly or not). These were dogs who had previously had recurrent UTIs, at least three times in the previous year. Six of the dogs were given a 2-week course of an antibiotic, and the other six were given a daily cranberry supplement for 6 months. All dogs had urine samples taken to look for infection once a month. None of the dogs developed a UTI during the 6-month monitoring period.
In the in vitro portion of the study, the urine from dogs given the cranberry supplement did not appear to have any inhibitory effects on bacterial growth. It did, however, seem to reduce the adhesion of these bacteria to canine kidney cells.
The design of this study is a little strange. There was no report of blinding or placebo control, and it is unclear if the subjects were randomly assigned to the two groups, so the risk of bias is difficult to assess. The strange thing, though, is that the comparison was between the cranberry supplement and a short course of an antibiotic rather than a placebo supplement. One would not expect a 2-week course of antibiotic to protect dogs from a urinary tract infection for 6 months. If resolving recurring UTIs were this easy, they would never happen since everyone would do this.
So in having no UTIs, the cranberry supplement group was no different from a group that received a treatment one wouldn’t expect to work, but which wasn’t a true placebo. It is very difficult to interpret this kind of comparison, but it certainly isn’t appropriate to say, it proves the supplement prevents infections, as the authors do. It is also not evidence that the supplement was safer than the short course of antibiotics, which the authors also claim. The lack of UTIs in all dogs may have been due to unexpected and dramatic effects from the antibiotics and the cranberry supplement which occurred despite these sorts of effects not having been seen in past studies. However, this may also reflect some other change in how these dogs were managed or simply the natural course of their disease. The purpose of controlled research is to help separate these possible causes, but this particular comparison doesn’t effectively accomplish that.
The second part of the experiment was consistent with past research showing that cranberry products do not inhibit bacterial growth but may reduce adhesion to urinary tract cells to some extent. Since kidney cells were used, it isn’t clear whether this effect would apply to cells in the bladder, where most urinary tract infections occur, or in actual living patients.
Bottom Line The in vitro portion of this study is consistent with existing research that suggests cranberry extracts may reduce the ability of some bacteria to stick to the lining of the urinary tract. This could theoretically help prevent some urinary tract infections, though clinical research in human patients suggests this doesn’t really work to a significant extent in living people.
The portion of the study looking at prevention of UTIs in actual dogs, unfortunately, doesn’t help establish what benefit, if any, this product might have. No infections occurred in any of the dogs regardless of the treatment they got, and the comparison was between 6months of daily use of the cranberry product and a 2-week course of an antibiotic, which seems unlikely to effectively prevent UTIs for 6 months. This raises questions about whether either of the treatments were really responsible for the lack of UTIs in the dogs or if this was the result of the Hawthorne effect or some other source of error.
Chronic kidney disease (CKD) is a common and important cause of illness in older cats. It is a frustrating disease for many reasons, especially the lack of a clear understanding of what causes CKD and the absence of a proven effective strategy for preventing it. Many theories have been advanced for what causes CKD, but the existing evidence is not conclusive for any of them.
Particularly popular among alternative medicine advocates is the notion that commercial diets and vaccines are important causes. The role of diet is not at all well-established. There are some studies suggesting that certain vaccines can induce the formation of antibodies against proteins in the kidneys of cats, but this has not yet been linked to actual disease. A new study evaluating risk factors for CKD in cats adds both some useful data and also some confusion to the subject.
This study involved signing up a large number of cats in the U.K. and collecting data about many aspects of their life and health. These cats were then followed over time, and differences between those who developed CKD and those who didn’t were evaluated. The results can’t be taken as proof that these differences are or are not direct causes of CKD, but it can help to identify potential risk factors.
Many different factors were evaluated, though of course the number of potentially relevant variables is enormous, so many were not evaluated which might still be important. As is common in this sort of study, most factors tested did not appear to differ statistically between the groups while a few did. The type of statistical significance testing used has come under growing criticism, with the major organization representing statisticians recently producing an unprecedented position statement suggesting this strategy be abandoned as misleading. This complicates interpretation of this and most other scientific studies employing this method. The results are presented in the two tables below.
The first shows statistically significant differences in 3 of 12 variables compared, with more cats developing kidney diseases being predominantly outdoors, vaccinated every 1-2 years (as opposed to >/= 3 years or never), and having moderate to severe dental diseases. The second table shows the results of linear regression analysis intended to confirm statistical association between some of these variables and kidney disease. This analysis found statistically significant differences in 4 of 16 variables, with the risk of kidney disease being greater with increased age, vaccination more often than every 3 years, and moderate or severe dental disease. In this analysis, being predominantly outdoors did not reach statistical significance as a risk factor.
The association between vaccination and kidney disease in this study will undoubtedly fuel opposition to vaccination of cats among some owners and veterinarians. It is an important observation since it does suggest that vaccination may be a risk factor for this common disease. However, even if this is true, it suggests that the current vaccination guidelines, which include vaccinating no more than every three years and considering even fewer vaccinations in cats who are indoors only and not exposed to vaccine-preventable disease, are probably sufficient to eliminate or dramatically reduce this risk.
Unfortunately, no additional information about which vaccines might play a role and which components in those vaccines might increase the risk of kidney disease is available in this study. Until the potential association between vaccination and kidney disease is confirmed and a specific understanding of the causal relationships involved in available, the only reasonable response to the possible risk is to adhere to current vaccination guidelines. More radical action, as is likely to be advocated by anti-vaccine advocates, is not justifiable and will likely only increase the risk of preventable infectious disease without protecting cats from kidney disease.
The other variables associated with kidney disease in this study were dental disease, age, and possibly outdoor lifestyle. Because dental disease is a chronic infectious and inflammatory condition, it is quite plausible that this disease might increase the risk of kidney problems and other health conditions. This emphasizes the importance of good oral hygiene for cats. Age has long been associated with kidney disease, and confirming this does not provide much in the way of actions we can take to reduce kidney disease risk. Finally, the finding that predominantly outdoor living may increase kidney disease risk is interesting, but it would certainly need to be confirmed before we could confidently include this risk among the many known to be associated with letting cat roam freely outside.
The failure to find associations between some variables and kidney disease is also worth considering. No association was seen, for example, between diet and kidney disease. It is often claimed that commercial diets, particularly dry kibble, increases kidney disease risk. However, in this study at least, such an association was not supported.
As always, there are a number of limitations to this study that require us to interpret the findings with caution. The use of p-values, and the risk of evaluating many factors and drawing conclusions about the few that show a significant association has already been mentioned.
The other most significant concern is that almost half (43%) of the cats initially enrolled in the study dropped out, and no information about their development of kidney disease is available. If these cats differ in any way from the cats who were followed up, this could completely change the results of the study. Additionally, of the cats who were followed to the end of the study, only 27 developed kidney disease, which is a pretty tiny sample for this kind of epidemiologic survey. By comparison, several similar studies in humans have included between 4000 and 9000 individuals.
There are a number of other ways in which the cats in this study might not fairly represent other populations at risk for kidney disease. Most were from urban areas in the U.K., and there may be significant genetic and environmental differences between these cats and other populations, such as suburban or rural cats in the U.S. Also, much of the information about diet and vaccination was obtained from owners and was limited in detail and reliability. All of these factors influence the extent to which these results can be generalized.
Overall, this study highlights a few variables that might or might not influence the risk of kidney disease. Further study will be necessary to confirm both the positive and negative findings and elucidate the details of any real risk factors to allow meaningful action to be taken to reduce the risk of this common and serious disease.
Evidence-based veterinary medicine (EBVM) is the explicit and deliberate integration of scientific research evidence with the circumstances and needs of individual patients to support clinical decision making. It requires that the practitioner be able to find and critically evaluate published reports of scientific research, and it therefore requires that these reports be accurate and reliable. But there is abundant evidence in human medicine that published research findings can often be misleading due to uncontrolled sources of error and bias. There is far less research concerning this problem in veterinary medicine, but a few studies have suggested the veterinary literature may suffer from similar weaknesses.
BIAS IN THE HUMAN MEDICAL LITERATURE
The design of scientific research studies is specifically intended to control for known sources of bias. The way that study subjects are selected and assigned to different groups, the way that researchers and subjects are blinded to the nature of these groups, and the way in which the resulting data are analyzed can all reduce error and lead to more reliable results. However, when studies are poorly designed, conducted, or reported, hidden bias can remain and the results can be misleading.
A clear source of potential bias is the funding for a study. In the human medical literature, funding source has been shown to be associated with outcomes in predictable ways that indicate bias is present. This bias can be reduced by strict adherence to standards of study design and conduct. However, evaluating the quality of these controls for bias requires that the methods used be adequately reported, and again the human medical literature shows that often insufficient information is present in many published reports to allow accurate assessment of the risk of funding bias.
The same is true for many sources of error in the human medical literature, such as publication bias, selection bias, information bias, confounding, and others. Often, the steps necessary to effectively minimize these sources of error cannot be assessed because the published reports provide insufficient information about how the studies were conducted. Some of the core aspects of study design and conduct, such as subject allocation and allocation concealment, blinding, and handling of subjects lost to follow up, are commonly not adequately reported or not properly implemented.
BIAS IN THE VETERINARY MEDICAL LITERATURE
There is less information about potential bias in the veterinary literature. The studies that have been done suggest that the quality of reporting is generally very low, and the risk of bias in individual studies is often be impossible to determine because of insufficient information. Several studies have found serious deficiencies in the reporting of randomization methods, blinding, and handling of losses to follow up, and serious problems with the quality of statistical methods employed in many study reports.
Even though risk of bias can be difficult or impossible to determine when reporting quality is low, some researchers have found that poor reporting is associated with a higher likelihood of positive results in published studies. This at least suggests that poor reporting may signal that the results biased in favor of the investigators preconceptions. Little research has been done on the subjects of funding and publication bias in the veterinary medical literature.
A SYSTEMATIC REVIEW OF VETERINARY CLINICAL TRIALS
I am currently conducting a systematic review of veterinary clinical trial reports intended to evaluate the quality of reporting and the risk of bias using instruments commonly used for this purpose in the human medical literature. Preliminary results suggest very poor quality of reporting generally, with some variation across species areas. Because of this poor reporting, it is difficult to assess the risk of bias in veterinary clinical trial reports. However, consistent with previous studies, these data do suggest that reporting quality is associated with the likelihood of positive outcomes, suggesting again that poorer quality studies are more likely to generate results consistent with the hypothesis under investigation. If this represents uncontrolled bias, then these results may be misleading.
This study does not consider the issue of publication bias since it is difficult to identify unpublished veterinary clinical trials. Unlike in human medical research, there is currently no widely used or mandatory registry for clinical trials, making it difficult to detect studies which may not be published due to negative results.
WHERE DO WE GO FROM HERE?
The deficiencies in the veterinary clinical trial literature are not a reason to abandon evidence-based medicine. EBM has had significant benefits in terms of patient care when implemented in the human medical field, and it is likely to be equally beneficial in veterinary medicine. However, the ability of veterinarians to practice EBVM is hampered by the limited reliability of published veterinary research, and steps should be undertaken to improve the reliability of the literature. Such steps could include mandatory reporting guidelines adopted by journals, funding agencies, and research institutions. This would improve the quality of reporting, which would make it easier to accurately assess the real risk of bias in published studies.
A number of organizations are also currently working towards the creation of a clinical trial registry for veterinary studies. This would greatly improve our ability to assess potential sources of bias, including publication bias and bias associated with losses to follow up in studies. If such a registry contained the original data collected, it would also help identify and deter questionable analytic or statistical practices.
Controlled scientific research is clearly superior to informal methods of evaluating diagnostic and therapeutic interventions because it minimizes the role of biases of all kinds. However, this function cannot be performed effectively if proper methods of study design and conduct are not employed. And the reliability of published evidence cannot be critically evaluated if the necessary information is not reported. A key feature, then, in the development of evidence-based veterinary medicine needs to be improvements in the quality and reliability of the veterinary medical literature.
Evidence-based veterinary medicine (EBVM) is the formal, explicit application of the philosophy and methods of science to generating understanding and making decisions in veterinary medicine. It is often associated with academic research and university or specialty practice. However, EBVM also provides a perspective and a set of behaviors veterinarian in clinical practice can employ to control bias, reduce errors, and manage information more efficiently in general practice as well. In this setting, where limited time and resources and the agendas of our clients constrain our actions, EBVM can facilitate better clinical decision making and improve patient care, aid in managing uncertainty, communicating effectively with clients, and establishing habits to facilitate ongoing learning and improvement throughout one’s career.
WHAT CAN EBVM DO FOR ME?
The main benefit of employing EBVM techniques is having better information on which to base our clinical practice. When there is good quality evidence to help us evaluate diagnostic and therapeutic interventions, EBVM helps us find this evidence and learn how to use it to inform our decisions. When, as is often the case, the evidence is poor in quality and quantity, understanding this helps us to avoid the risks of unjustified certainty and be mindful of the need for flexibility and lifelong learning in clinical practice.
Better information, and more informed decision making, leads to better patient care. There is evidence from human medicine that the implementation of evidence-based clinical practice guidelines and other EBM tools improves patient outcomes, and this is the goal of EBVM as well.
Finally, EBVM can help practitioners meet our ethical obligations to patients and clients. We have a duty to our patients to provide the best care possible, and EBVM facilitates this. We also have a duty to provide truly informed consent to our clients. Only by understanding the evidence behind our recommendations, and having a clear view of the degree of uncertainty present, can we effectively guide clients in making decisions for their animals.
THE STEPS OF EVIDENCE-BASED PRACTICE Evidence-based practice involves following a set of explicit steps to integrate formal scientific research information with the individual circumstances of each case to facilitate decision making. The busy practitioner will clearly not be able to execute each step for every problem in every case, nor is this necessary. But by regularly employing the EBVM process, we build and maintain a knowledge base that informs our decisions.
Of course, every veterinary clinician already has extensive knowledge and opinions that inform his or her practice. However, without EBVM, our knowledge base is haphazard and uncritically derived from sources of unknown or low reliability. EBVM allows us to have greater confidence in the knowledge we rely on when making recommendations for individual patients.
These are the basic steps of EBVM:
Ask useful questions
Find relevant evidence
Assess the value of the evidence
Draw a conclusion
Assign a level of confidence to your conclusion
Asking Useful Questions Vague or overly broad questions impede effective use of research evidence in informing clinical practices. “Does drug X work?” or “What should I do about disease Y?” are not questions that are likely to lead to the recovery of useful information from published research. There are a number of schemes for constructing questions the scientific literature can help answer. One of the easiest is the PICO scheme.
P– Patient, Problem Define clearly the patient in terms of signalment, health status, and other factors relevant to the treatment, diagnostic test, or other intervention you are considering. Also clearly and narrowly define the problem and any relevant comorbidities. This is a routine part of good clinical practice and so does not represent “extra work” when employed as part of the EBVM process.
I– Intervention Be specific about what you are considering doing, what test, drug, procedure, or other intervention you need information about.
C– Comparator What might you do instead of the intervention you are considering? Nothing is done in isolation, and the value of most of our interventions can only be measured relative to the alternatives. Always remember that educating the client, rather than selling a product or procedure, should often be considered as an alternative to any intervention you are contemplating.
O– Outcome What is the goal of doing something? What, in particular, does the client wish to accomplish. Being clear and explicit, with yourself and the client, about what you are trying to achieve (cure, extended life, improved performance, decreased discomfort, etc.) is essentially in evidence-based practice.
FIND RELEVANT EVIDENCE
Experienced clinicians typically have opinions on the value of most interventions they routinely consider. Unfortunately, we rarely know where those opinions originally came from or how consistent they are with the current best scientific evidence. And given the constraints of time and resources, practitioners will rarely have the ability to find and critically evaluate all the primary research studies relevant to a particular question. Fortunately, there are sources of evidence that can provide reliable guidance in an efficient, practical manner.
The best EBVM resource for busy clinicians is the evidence-based clinical practice guidelines. These are comprehensive evaluations of the research in a general subject area that explicitly and transparently identify the relevant evidence and the quality of that evidence and make recommendations with clear disclosure of the level of confidence one should place in those recommendations based on the evidence.
Sadly, many guidelines produced in veterinary medicine are not evidence based but opinion-based (so-called GOBSAT or “Good Old Boys Sat At a Table” guidelines). These are no more reliable than any other form of expert opinion. Excellent examples of truly evidence-based guidelines are those of the RECOVER Initiative for small animal CPR and the guidelines produced by the International Task Force for Canine Atopic Dermatitis.
After evidence-based guidelines, the next most useful resources are systematic reviews and critically-appraised topics (CATs). These are more focused but still explicit and transparent reviews of the available evidence on specific topics. Systematic reviews can be identified by searching the VetSRev database, a free online resource produced by the Centre for Evidence-based Veterinary Medicine (CEVM) at the University of Nottingham. Unfortunately, getting full-text copies of these reviews can be challenging for vets not at universities, but there are a number of options depending on where one practices.
Critically appraised topics are also produced by CEVM and freely available on the web as BestBetsforVets. There are a number of other free CAT resources, including the Banfield Applied Research and Knowledge web site.
Finally, primary research studies are a useful source of guidance for clinicians, though they take more effort and expertise to find and critically evaluate.
ASSESS THE VALUE OF THE EVIDENCE
The most challenging part of the EBVM process for vets in practice is critical appraisal, learning to identify important limitations in published research study that affect how confident we can be in the conclusions and how relevant they are to our patients. There are resources available to teach these skills, and hopefully this will become more common in veterinary colleges, but for most practitioners pre-appraised evidence, such as guidelines and systematic reviews, will be more useful.
The clinician still has an important role, however, in determining the relevance of research evidence to individual patients. The details of a patient’s medical condition, the values, goals, and resources of the owner, and the expertise and resources available to the veterinarian all determine the degree to which a particular conclusion based on formal research is applicable to a given patient. The role of EBVM is not to replace clinician judgment with automatic reliance on published research but to ensure the clinician has the best available information and understands clearly what is known and not known when tailoring the evidence to the needs of individual animals.
DRAW A CONCLUSION
Ultimately, the job of a veterinarian is to guide the client in making decisions about care for their animals. When the clinician is aware of the existing evidence and its limitations and clearly appreciates the degree of uncertainty, then he or she can best help the client to understand their options. Making evidence-informed decisions and clearly communicating with clients about the needs and choices for their animal is the core of clinical veterinary medicine, and this is what the tools and methods of EBVM exist to support.
ASSIGN A LEVEL OF CONFIDENCE TO YOUR CONCLUSIONS
Often, the relevant research evidence is incomplete or flawed, and sometimes there is little or no such evidence applicable to a given patient’s needs. EBVM is still useful in this situation, because it allows us to clearly, systematically identify and communicate the uncertainty inherent in our work.
It is also important that we openly discuss with clients our use of evidence to inform our recommendations. Research has suggested that clients want to be told about the uncertainties involved in the treatment of their animals, and that discussing this does not reduce their confidence in their veterinarians. Clients also identify truthfulness as their highest priority in communication with their vet. By explicitly discussing our process in identifying and evaluating relevant evidence, we enhance our clients’ understanding of the role we play, and we help them to appreciate the value of our expertise, not only the products and procedures we sell.
EBVM AND THE GENERAL PRACTITIONER
The job of the general practitioner is to be informed about the research evidence relevant to their patients’ needs and to think critically about this evidence and the uncertainty it contains. It is also the role of practitioners to communicate clearly with clients about this information and guide them in making informed decisions. Ideally, general practitioners can also contribute by sharing what they learn in applying the EBVM process. Critically appraising individual studies or synthesizing the literature on particular questions will create useful information that can then be shared with colleagues.
Properly applied and with adequate support, EBVM can enhance the quality of information supporting the decisions and recommendations of vets in clinical practice. This not only reduces stress and wasted effort for veterinarians but improves client communication and patient care.
Experienced clinicians often have an enormous knowledge base about the health problems their patients present with and the available diagnostic and therapeutic options. This knowledge is built over time from a variety of sources: basic pathophysiology and clinical information learned in school; practice tips and pearls imparted by professors and speakers at continuing education meetings; review articles and primary research papers in veterinary journals; textbooks; advice from mentors and colleagues in practice; and of course clinical experience with previous cases.
This knowledge base smoothly and efficiently informs the day-to-day activities of clinical practice. When cases with familiar features are seen, the appropriate diagnostic and treatment steps often come to mind automatically, or with minimal prodding. Unlike students and new graduates, experienced clinicians often have little sense of dredging up facts committed to memory and more of a sense of simply knowing things. One of the hallmarks of expertise is that the collating of observations and relevant knowledge into a coherent picture of the problem and a plan become less deliberate and more automatic with time.1
While this process, which is a universal and automatic feature of how the human brain functions, leads to greater efficiency than the explicit, conscious use of algorithms and reference sources employed by less experienced practitioners, it has a number of potential limitations. One problem, for example, is that the knowledge one relies on often can no longer be connected to its original source. We often simply know things without being aware of how we came to know them. This limits our ability to judge the reliability of the source of our knowledge. In fact, such established, automatic knowing often generates a sense of certainty greater than that which accompanies deliberately seeking and finding information.2 We are more likely to trust what we already know, even if we don’t remember where we learned it, than we are to trust what we have just discovered after searching a trustworthy source of information.
There are also a large number of well-characterized cognitive biases and sources of errors inherent in how our brains acquire, process, store, and utilize information that can lead us astray.3 These are more likely to create error when our reasoning is automatic rather than deliberate, as it necessarily must be in an efficient clinical environment that is not devoted primarily to teaching.
One of the major functions of evidence-based veterinary medicine (EBVM) is to provide tools and resources to make the knowledge base we employ more reliable. This includes generating better quality information through research and facilitating the integration of that information into clinical decision making. When the relevant evidence is of high quality, this can add confidence to our decisions.
More commonly, when the evidence has significant limitations, we may end up with less confidence in our knowledge than we would have without an explicit evaluation of the evidence. However, this is not as undesirable an outcome as it may appear. A clear, accurate understanding of the uncertainty associated with a particular practice protects us, and our patients, from the dangers of acting with unjustified confidence. We are more likely to weigh thoughtfully the risks and benefits of action in the context of an individual case when we understand the degree of uncertainty about our ability to predict or manipulate the patient’s condition.
Being clear about the sources of our knowledge, and the appropriate level of confidence to have in them, also aids in fulfilling our duty to provide clients with informed consent. Surveys of veterinary clients have shown that they value truthfulness highly in the information we provide to them, and that they want to be made aware of the uncertainties involved in the treatment of their animals.4-5 Only if we understand the reliability and limitations of the information we employ in making our recommendations can we give clients the knowledge and guidance they need to make informed choices.
The purpose of these lectures is to examine some widespread or long-standing beliefs and practices in small animal medicine and assess their evidentiary foundations. In some cases, this may clearly validate or invalidate these beliefs. In most cases, however, such an exploration will likely not lead to greater certainty but to a clearer understanding of the degree of uncertainty associated with these beliefs. Hopefully, this will be useful in making clinical decisions and in communicating with clients. The exercise may also be useful in illustrating how to make use of the research literature in establishing and maintaining the knowledge base that informs one’s clinical practice.
REFERENCES
Benner, P. From novice to expert. Amer J Nursing, 1982; 82(3):402-7.
Burton, R. On Being Certain: Believing You’re Right Even When You’re Not. New York: St. Martin’s Press. 2008
McKenzie, BA. Veterinary clinical decision-making: cognitive biases, external constraints, and strategies for improvement. J Amer Vet Med Assoc. 2014;244(3):271-276.
Mellanby RJ, Crisp J, De Palma G, et al. Perceptions of veterinarians and clients to expressions of clinical uncertainty. J Small Anim Pract 2007;48:26–31.
Stoewen DL, et al. (2014) Qualitative study of the information expectations of clients accessing oncology care at a tertiary referral center for dogs with life-limiting cancer. J Am Vet Med Assoc. 2014;245(7):773-83.
I have collected here the various posts written about my experiences taking the Medical Acupuncture for Veterinarians training course. This provided an opportunity to take a detailed look at the claims and evidence made for acupuncture outside of the realm of Chinese folk medicine.
WHAT IS CLINICAL AUDIT? “Clinical audit is a process used by health professionals to assess, evaluate and improve patient care…Clinical audits can be used to compare current practice with the best available evidence. It provides a methodology to assess if the best evidence-based medicine is being applied within the practice.”1
Clinical Auditing is a quality improvement process in clinical practice that seeks to establish guidelines for dealing with particular problems, based on documented evidence when it is available, measuring the effectiveness of these guidelines once they have been put into effect, and modifying them as appropriate. It should be an ongoing upwards spiral of appraisal and improvement.2
WHY SHOULD I CONDUCT CLINICAL AUDITS? The most obvious and significant reason to conduct clinical audits is to improve patient care. Though it is a difficult idea to prove empirically, it is likely true that formal processes to evaluate patient outcomes, implement evidence-based changes in clinical practices, and then assess the impact of these changes on patient outcomes should improve patient care. There is some research evidence to support this concept, though the effectiveness of clinical audit depends on many factors, such as the baseline level of performance, the details of how changes are implemented, and others.3
In addition, clinical audit can contribute to job satisfaction for all members of the veterinary healthcare team. Taking active steps to assess the effectiveness of one’s practices and implement change and then seeing the results of those efforts can give veterinarians and veterinary nurses a greater enthusiasm for their work, and a sense of confidence in their clinical decisions.
Clinical audit can also be used as a means to strengthen client confidence in a practice and reassure clients about the risks of specific interventions. Knowing that there is an established and ongoing process of quality control in place promotes trust in the doctors and the practice. And if clients have concerns about the risks of surgery or other interventions, clinical audit enables us to provide them with specific and relevant data to support fully informed consent and reassure them about the procedure and our commitment to safe and effective care
TYPES OF CLINICAL AUDIT There are two basic types of clinical audit, though the specific goals and procedures are tailored to each setting and question. Standards-based audits are those which compare current practice and results to some designated standard. The standards can be goals set specifically for the practice or derived from published clinical guidelines or other external, evidence-based sources.
A standards-based audit could evaluate how often antibiotics prescribed for urinary tract infections turned out to be appropriate based on subsequent culture and sensitivity results, with an eye to adjusting empirical antibiotic choices if necessary. Or such an audit could be focused on ensuring a certain percentage of newly-diagnosed cases of feline chronic kidney disease were fully staged according to the published IRIS guidelines.5 Or one could use a process audit to evaluate patient outcomes, such as the proportion of spay surgeries experiencing seroma or other complications, with the goal of optimizing practices to minimize these.
Critical Incident or Significant Event audits are structured discussions that occur in response to some identified error or process failure or simply an undesired outcome. Such events can be clinical, such as an anesthetic death or patient escape in the hospital, or they can be procedural, such as a failure to submit a urine sample for culture or to report biopsy results in a timely manner.
The goal of a significant event audit is not to assign blame but to determine if any feature of hospital policy or procedure might have increased the risk of such events and how this risk might be reduced through any process changes.
HOW DOES CLINICAL AUDIT WORK? The specific steps in a clinical audit will be somewhat different in every setting. However, there are some general principles that apply broadly to the audit process. For audits to result in meaningful, effective change in practices, the entire healthcare team must be convinced of the value and legitimacy of the process. This requires participation and input by all team members, rather than an authoritarian “top-down” approach. Having clear and transparent procedures agreed upon in advance for how audits are to be conducted and a regular, consistent execution of these procedures will facilitate creating an institutional culture in which audits are seen as a routine and necessary part of patient care.
The general steps of the audit process are illustrated in Figure 1.
Figure 1. The process of clinical audit (from Dunn, 2012)3
The process begins by identifying a question or area of concern to investigate. All members of the team should be invited to offer questions for consideration and an open discussion of these and how to structure and prioritize this should be held involve the entire team. Individuals or a small team responsible for a particular audit should then identify guidelines or standards to be used as benchmarks and develop a plan for collecting appropriate data. This plan should again be discussed with all members of the team who will be involved to make sure it is appropriate and not unduly complex or burdensome.
Once the data is collected, it should be analyzed to compare the results to the designated standards. Audits are not intended as clinical research projects, and no statistical analysis is necessary or appropriate. The goal is to compare the results in the practice setting clearly and directly with the designated guideline or standard in a way that identifies potential areas for improvement.
Since the overall purpose of audits is to facilitate improvement, when an audit identifies practices that do not meet the designated standard, the entire team should again be involved in formulating a plan for implementing changes to improve performance. Once these are well established, a repeat of the audit cycle can be sued to assess whether the changes have had the desired effect.
Though audits are not intended to be a form of controlled clinical research, they do have the potential to produce data useful to those outside the practice in which they are conducted. The results are specific to the environment in which the audit was conducted and so cannot necessarily be generalized to other settings. However, in the evidence-poor environment of veterinary medicine, clinical audit can at least suggest hypotheses about best practices or the impact of clinical practice guidelines and provide at least rough estimates of the frequency of some outcomes in various settings. Therefore, sharing the results of clinical audits is to be encouraged. This also has the added benefit of making clients aware of the active quality improvement processes in place, helping to build confidence in the practice.
BARRIERS TO CLINICAL AUDIT There are a number of barriers to conducting clinical audits. The biggest in the U.S. is the lack of awareness of the audit process. Clinical audit is well-established in the human healthcare system in various forms. And in the U.K., veterinary practices are required to have some formal mechanism for assessing outcomes and improving performance, which can be satisfied by a clinical audit mechanism. However, there is not widespread awareness of the value or methods of clinical audit among veterinarians in the U.S.
Other practical barriers include time and other resources, resistance to change from team members, and concerns about the impact of identifying errors or undesirable outcomes on employee performance evaluations or client confidence. All of these are limitations that have been successfully addressed in the implementation of clinical audit in human medicine and that can be overcome in the use of this valuable tool in veterinary medicine.
RESOURCES The Royal College of Veterinary Surgeons (RCVS) Knowledge group has published a useful Clinical Audit Toolkit that is freely available online. The references below also include several useful guides to the implementation of clinical audit in veterinary practice.2,4,6
REFERENCES
Cockroft P, Holmes M. Handbook of evidence-based veterinary medicine. Oxford, England: Blackwell Publishing, 2003
Viner, B. Introducing clinical audit into veterinary practice.PhD. dissertation, Middlesex University, London, England. 2006.
Jamtvedt G, Young JM, Kristoffersen DT,O’BrienMA,Oxman AD. Audit and feedback: effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2006;(2):CD000259.
Dunn, J. Clinical audit: A tool in defense of clinical standards. In Practice. 2012;34:167-169.
