I recently gave a lecture at the Western Veterinary Conference called “What You Know that Ain’t Necessarily So.” The purpose of this was to take some common or controversial beliefs and practices in veterinary medicine and discuss the scientific evidence pertaining to these. This was not intended as a definitive, “final word” on these subjects, but as an illustration of how weak and problematic the evidence often is even behind widely held beliefs. In some cases, these practices or ideas may actually be valid, but without good quality scientific evidence, we should always be cautious and skeptical about them.

Eventually, I will post recordings of the presentations themselves, but for now I am posting a summary of each topic.

Each starts with a focused clinical question using the PICO format.

P– Patient, Problem Define clearly the patient in terms of signalment, health status, and other factors relevant to the treatment, diagnostic test, or other intervention you are considering. Also clearly and narrowly define the problem and any relevant comorbidities. This is a routine part of good clinical practice and so does not represent “extra work” when employed as part of the EBVM process.

I– Intervention Be specific about what you are considering doing, what test, drug, procedure, or other intervention you need information about.

C– Comparator What might you do instead of the intervention you are considering? Nothing is done in isolation, and the value of most of our interventions can only be measured relative to the alternatives. Always remember that educating the client, rather than selling a product or procedure, should often be considered as an alternative to any intervention you are contemplating.

O– Outcome What is the goal of doing something? What, in particular, does the client wish to accomplish. Being clear and explicit, with yourself and the client, about what you are trying to achieve (cure, extended life, improved performance, decreased discomfort, etc.) is essentially in evidence-based practice.

This is then followed by a summary of the evidence available at each of the levels in the following pyramid (which is a pragmatic reinterpretation of the classical pyramid of evidence that is a bit more useful for general practice veterinarians).

Finally, I list the Bottom Line, which is my interpretation of the evidence.

Neutering and Mammary Neoplasia in Dogs

1. Clinical question

P- female dogs

I- neutering (timing)

C- remaining intact (timing)

O- incidence/mortality mammary neoplasia

What is Risk in Intact Females?

• 53.3% (varies by breed and age: 4-35/1,000/yr)
  (Moe, 2001)

• 1% 6yrs, 6% 8yrs, 13% 10yrs
• 111/10,000 DYAR (5-319/10,000 DYAR)
  (Egenvall, 2002)

• 205 tumors/100,000 dogs/year SI
  (Dobson, 2002)

2. Synthetic Veterinary Literature

a. One systematic review:

13 peer-reviewed journal articles in English

     9 high risk of bias

     4 moderate risk of bias

          1 found protective association

         2 found no association

         1 found“some protective effect” but no numbers

(Beauvais, 2012)

b. One critically appraised topic

Best Bets for Vets Age at neutering and mammary tumours in bitches

Spaying bitches before the first or second season, or before the age of 2.5 years, may be associated with a reduced risk of developing malignant mammary tumours later in life…However, the evidence is relatively weak, and this should be taken into account alongside other considerations when recommending whether and when to neuter.

3.  Primary Veterinary Literature

• Strong protective effect
• No confidence interval or p-value
• Matched cases/controls, matched analysis?
• Cases/controls from different time periods
• Only for cases with histopathology
• Previous hormone use?

(Schneider, 1969)

• “inconsistent although some protective effect”
• No quantitative assessment
• No control for age, previous hormone use
• Research Beagles
• Neutered at 10-12 years of age
• Primary purpose to evaluate Rad Tx

(Bruenger, 1994)

• No association (neutering & any type of mammary mass)
• Age at neutering not reported
• No control for age, breed, previous hormone use
• Only for cases with histopatholog
• Primary purpose diet and mammary tumors

(Pérez Alenza 1998)

• No association (neutering & proportion of submitted tumor samples neoplastic)
• Age and age at neutering not reported
• No control for age, previous hormone use
• Only for cases with histopathology
• No confidence intervals

(Richards, 2001)

• No association between age at neutering and neoplasia
• All neutered 6 weeks-12 months
• Mammary neoplasia not specifically addressed
• Shelter dogs
• Underpowered for uncommon outcomes

(Spain, 2004)

4. Human Literature- Primary

> 66,000 women studied, observational

The risk for breast cancer was reduced by 27% among women who had hysterectomy and BSO before 45 years of age, and by 20% among those who had simple hysterectomy before age 45 years.

(Gaudet, 2014)

~4,500 women studied, case/control

Bilateral ovariectomy was associated with reduced breast cancer risk overall (odds ratio (OR) = 0.59, 95% confidence interval (CI): 0.50, 0.69) and among women <45 years of age (ORs ranged from 0.31 to 0.52), but not among those who were older at surgery.

(Press, 2011)

Bottom Line–

• Available evidence suggests neutering reduces mammary cancer risk
• Earlier neutering may be more protective
• Great variation with breed, age, other factors
• Existing evidence is very weak
• Strong or numeric claims not justified

Reference

Beauvais W, Cardwell JM, Brodbelt DC. The effect of neutering on the risk of mammary tumours in dogs–a systematic review. J Small Anim Pract. 2012 Jun;53(6):314-22.

Bruenger, FW, et al. Occurrence of mammary-tumors in beagles given RA-226. Radiation Research 1994;138:423-434.

Dobson JM, Samuel S, Milstein H, Rogers K, Wood JL. Canine neoplasia in the UK: estimates of incidence from a population of insured dogs. Journal of Small Animal Practice 2002;43(6);240-6.

Egenvall A, Bonnett BN, Ohagen P, Olson P, Hedhammar A, von Euler H. Incidence of and survival after mammary tumors in a population of over 80,000 insured female dogs in Sweden from 1995 to 2002. Preventative Veterinary Medicine 2002;69:109-27.

Gaudet MM, et al. Obstet Gynecol. Oophorectomy and Hysterectomy and Cancer Incidence in the Cancer Prevention Study-II Nutrition Cohort. 2014:123;1247-1255.

Moe L. Population-based incidence of mammary tumors in some dog breeds. Journal of Reproduction and Fertility 2001;57:439-43.

Pérez Alenza D. et al. Relation between habitual diet and canine mammary tumors in a case-control study. Journal of Veterinary Internal Medicine. 1998;12;132-139.

Press DJ, et al. Breast cancer risk and ovariectomy, hysterectomy, and tubal sterilization in the Women’s Contraceptive and Reproductive Experiences Study. Am J Epidemiol. 2011;173(1):38-47.

Richards HG, et al. An epidemiological analysis of a canine-biopsies database compiled by a diagnostic histopathology service. Preventive Veterinary Medicine. 2001;51:125-136.

Schneider R, Dorn CR, Taylor DO. Factors influencing canine mammary cancer development and postsurgical survival. Journal of the National Cancer Institute 1969;43:1249-61.

Spain CV, Scarlett JM, Houpt KA. Long-term risks and benefits of early-age gonadectomy in dogs. Journal of the American Veterinary Medical Association 2004;224(3):380-7.

Without question, the most frequent comment I get in response to my articles is that people feel some of the therapies I write about must be effective regardless of the scientific evidence because they have had personal experiences that suggest this. The experience of having used a therapy and seen an improvement is very powerful psychologically, and it makes us very confident that the therapy we used caused the improvement we saw. Unfortunately, the evidence that this kind of conclusion is not reliable is overwhelming. This kind of thinking is so common, and so untrustworthy, it constitutes a unique ogical fallacy, the post hoc ergo propter hoc, or “false cause” fallacy.

Reality is complex, and our minds seek simple, direct causal explanations and satisfying narratives to explain things. Sometimes, of course, these explanations are true. But they are false much more often than we realize. The dismissal of anecdotal evidence by scientists isn’t casual, and it isn’t based on the idea that people who have anecdotal experiences must be stupid. It is based on centuries of study of the human mind, including decades of controlled research in cognitive psychology that shows things simply aren’t always as they seem.

John Stuart Mill described eloquently the problem that while we all realize we are imperfect and acknowledge in general terms that we can be wrong, we are very, very reluctant to ever admit we are wrong about any specific belief:

Unfortunately for the good sense of mankind, the fact of their fallibility is far from carrying the weight in their practical judgment, which is always allowed to it in theory; for while everyone knows himself to be fallible, few think it necessary to take any precautions against their own fallibility, or admit the supposition that any opinion, of which they feel very certain, may be one of the examples of the error to which they acknowledge themselves to be liable.

This helps to explain why I so often have to repeat, to pet owners and other veterinarians alike, that just because they have seen something appear to work with their own eyes, that isn’t really a good reason to believe it actually does work without supporting controlled scientific evidence.

A recent article on the subject both explains the problem, and discusses some of the possible solutions.

Your Brain is Primed to Reach False Conclusions by Christie Aschwanden

Here’s her conclusion:

With a lot of evidence that erroneous beliefs aren’t easily overturned, and when they’re tinged with emotion, forget about it. Explaining the science and helping people understand it are only the first steps. If you want someone to accept information that contradicts what they already know, you have to find a story they can buy into. That requires bridging the narrative they’ve already constructed to a new one that is both true and allows them to remain the kind of person they believe themselves to be.

The good news is that the fallibility of uncontrolled personal observation is well-documented, and this has been known for hundreds of years. This means it should be possible to inoculate people against excessive confidence in their own experiences early, teaching critical and skeptical thinking early in school, even before formal teaching of scientific facts. The bad news, however, is that once people reach a conclusion based on anecdotal experience, facts are not very effective at challenging that conclusion. Being given evidence that we are wrong tends to strengthen our false beliefs and impel us to build more and stronger arguments to support them.

Sadly, simple presentation of facts isn’t enough. People need to be led to reconsider their opinions through arguments that speak to their emotions and their core values, not simply their intellect. This is certainly more challenging than simply presenting the facts, especially for scientists who tend to think in terms of objective evidence and are suspicious or arguments that appeal primarily to emotions of beliefs rather than facts. And, of course, the biggest challenge in changing minds about scientific topics is that the stronger one’s belief, the less willing one is to seriously consider alternative explanations. The very people who most need to read about why anecdotes can’t be trusted won’t.

Despite all that, since I have to make the same argument over and over again, I have collected a few resources on this subject to which I refer anyone interested and open-minded enough to consider them:

Medical Miracles: Should We Believe?

Testimonials Lie

Alternative medicine and placebo effects in pets

Placebo effects in epileptic dogs

Medical Practices Once Widely Accepted that Proved Ineffective or Harmful when Studied Scientifically

Introduction

Yes, I know the title is a misquotation of the original line, but I couldn’t resist.

The idea that music might be useful in calming animals or otherwise influencing their mood and behavior is not new, but there seems to have been increased interest in it lately in veterinary medicine, specifically in the use of music to reduce anxiety and stress in hospitalized animals. Given the profound effect music has on human emotions, it is reasonable to ask the question, could animals experience the same kind of emotional effects and could music help soothe them in times of stress? As a veterinarian, a musician (well, I play mandolin a bit, anyway), and a former primatologist working in environmental enrichment, I find this an intriguing idea.

I’m sure, though, that it will come as no surprise to regular readers that I have some skepticism about the idea as well. I know from my own behavior research work and clinical veterinary practice that it is extremely difficult to consistently and objectively define and measure stress and mood in non-human animals. I also know that it is all too easy to assume that the world seems much the same to our animal companions as it seems to us, and that they share our likes and dislikes. Finally, I also know that nothing in medicine is free, and if music has meaningful benefits for our pets and patients, it likely has some risks as well.

Humans are unique in the extent to which they deliberately create patterns of sounds, often using tools made for that purpose, with the specific intent of inducing emotional responses in others of our species. Arguably, many species create music of a sort. Birds, humpback whales, and others, can create quite complex and engaging sound patterns. But I think it is fair to say that these differ in many ways from the type of music humans create and the purposes these sounds serve. So I think it fair to be cautious in assuming other animals will perceive or respond to human music as we do.

And as we all know, music is incredible in its variety, with dramatic differences in musical styles and tastes between cultures, generations, and individuals. What I find energizing or entertaining, my wife may find grating and annoying. So again, we should be careful in making confident generalizations about what other animals may like or dislike, or whether music will have the kinds of effects on them we anticipate. Still, the relationship between music and behavior, physical stress, and even health is worth exploring.

What’s the Evidence?

There are quite a few studies looking at the effect of various sounds, including music, on a variety of animal species. Many of these involve laboratory animals, such as rodents and primates. There are also a few looking at livestock species and dogs. A nice narrative review of this literature was published not long ago in the journal Lab Animal:

Alworth LC, Buerkle SC. The effects of music on animal physiology, behavior and welfare.  Lab Anim (NY). 2013 Feb;42(2):54-61.

The clearest answer that comes out of this review to the question, “Does music have beneficial effects on animals?” is, “It depends.” A wide variety of physiologic and behavioral responses to various types of musical stimuli have been observed, and they seem to vary not only from species to species but with the type of music and even with the particular study. And while many of the results are consistent with the investigator’s expectations, there are plenty of findings that were surprising. Music can have effects that might be beneficial or harmful or, in some cases, it might have no measurable effects at all.

As an example of the variety of results, one set of studies showed that playing Mozart reduced blood pressure in hypertensive rats. However, other studies found no effect of Mozart on blood pressure. This may have been due to the use of different specific pieces of music, to differ methods of measuring blood pressure, to differences in the types of rats used, or to any of a hundred other differences between the studies.

Similarly, different types of music seem to have different effects, but the differences don’t always form a clear, predictable pattern. In general, “quieter” types of music, such as classical or “New Age” music seem to reduce physiologic and behavioral responses associated with stress, and other genres either have no effect or may even worsen these measures. However, some studies looking at classical music find no effects, or effects for some composers and not others, and other studies find positive effects from other genres, or even white noise.

Finally, apart from the variety in results of different studies, there are serious problems with the way the effects of music are measured and interpreted. Most of the studies in dogs, for example, contain few of the usual controls for bias and error so important in making scientific research results reliable. Subjects are often chosen and allocated to different groups by convenience rather than randomly, and the investigators observing and recording behaviors are rarely blinded to the music condition.

If you think classical music is more likely to calm dogs than country music and you watch dogs and note their behavior while you and they listen to each kind of music, it is quite likely you will unconsciously interpret what you see and what you think it means through the filter of your own expectations. This is why, after all, controlled scientific research is necessary in the first place.

The authors of this review provide what I think is a very fair summary of what we know so far:

The studies reviewed in this paper show that music can affect the physiology, production, and behavior of multiple non-human species and provide evidence that some effects of music exposure are similar between humans and animals. The studies identify the potential for music to benefit the welfare of animals in many captive environments…As the reviewed studies show, however, the effects of music exposure are not the same for all music styles, animal species, or situations, and playing music for captive species has the potential to increase stress or impair welfare. Therefore, each situation should be considered individually and the outcome evaluated for positive and negative effects.

Bottom Line

The best summary of what we know now about the effects of music on animals comes again from the review cited above:

The studies described above support the idea that physiological and neurochemical changes can result from music exposure. Music’s ability to improve learning and memory and diminish stress suggests that music can result in positive physiologic effects. Other studies, however, suggest that music has no observable effect or can have a detrimental effect on animals. There is not yet sufficient evidence to suggest that animals react physically to music in a way that can be manipulated predictably and consistently.

This means that while it is very likely some kinds of music can be beneficial to our pets and veterinary patients under some circumstances, the devil is, as usual, in the details, and we don’t know much about the details yet. It is reasonable to experiment with music in veterinary environments, especially with more quiet, instrumental genres, but we must try to develop objective measures of the effect to be sure we are not imply wasting our effort or, even worse, actually adding stress for our patients.

As a proponent of evidence-based medicine (EBM), I often emphasize that in the absence of good-quality controlled research evidence, we cannot reliably assess the safety or efficacy of medical treatments. And as a skeptic, I am often forced to clarify that skepticism is not the automatic rejection of the unfamiliar but the position that conclusions about whether specific claims are true or false should be based on reliable evidence. Again, in the absence of such evidence, I do not assume claims to be false–I classify them as unproven. A popular aphorism to express this position is, “The absence of evidence is not evidence of absence.”

However, despite a degree of truth in it, this aphorism is frequently and widely misused. Proponents of unproven therapies often use it to suggest that such therapies should be given the benefit of the doubt, assumed to be innocent (that is, safe and effective) until proven guilty (that is unsafe or ineffective). When I point out on this blog that specific practices have no good evidence to support them (and no, anecdotes and testimonials do not count as “good” evidence), the response from advocates of these practices is often, “You can’t prove X doesn’t work!” That entirely misses the point in several ways.

For one thing, the burden of proof is always on those making a claim, not those asking for evidence to support it. If I say there is an invisible, man-eating dragon in my garage and you doubt me, it is not your job to conclusively prove the dragon doesn’t exist. There is no need for you to take this, or any other claim about the world, seriously until the person making the claim provides evidence for it.

This is especially true for claims that contain within them assumptions that violate well-established facts. Since invisibility and dragons have never been proven to exist, and there is good reason to think they do not, my claim is not only unproven but unlikely. In statistical terms, my hypothesis has a low prior probability.

Prior probability is, in a sense, an exception to the principle that “unproven” does not mean “false.” It is technically true that my guard-dragon is only unproven, not definitively disproven. But in practical terms, the prior probability of its existence is so low that it makes more sense to behave as if it does not exist than to behave as if it does. Would you choose never to enter my garage no matter what just in case the dragon might eat you? Would this be sensible?

The same principle applies in medicine. Many therapies, both conventional and alternative, are unproven in the sense that there is not robust research evidence to characterize their safety and efficacy in all possible situations. One misconception about evidence-based medicine is that such an absence or weakness of evidence means we are to refrain from making any decision about these therapies. EBM does not require that we stand idle with our hands in our pockets whenever there is no systematic review or large, high-quality clinical trial evidence available concerning the therapies we are considering using. We must quantify and acknowledge the uncertainty associated with weak evidence, but this is only one part of the job of balancing the need to intervene with the degree of uncertainty about the consequences of our interventions.

Unfortunately, some proponents of EBM do fall a little ways into this trap, often concluding that in the absence of perfect evidence, “no conclusion can be drawn” and “more evidence is needed.” More evidence may often be desirable, but a clinician working with actual patients must always draw a conclusion, however tentative. That is our job, to guide and care for patients as best we can using the evidence we have, not the evidence we wish we had.

On the other hand, proponents of implausible or unproven therapies often make the opposite error, assuming that the absence of evidence frees them to do as they like without taking into account the uncertainty of not having good scientific evidence. While clinicians may often be forced to rely on clinical experience alone, we should never forget how deeply unreliable a guide this is. As I’ve pointed out before, the three most dangerous words in medicine are “In my experience….” The absence of evidence should not reassure us that anecdotes, personal experience, historical or cultural tradition or any other information with low reliability and high risk of bias can be sufficient to support or recommend a therapy. In medicine, “unproven” may not mean “false,” but it absolutely means “risky!”

Because there is always a chance of doing harm, of making a patient worse when we intervene, it is incumbent on doctors to be wary of interventions with a great deal of uncertainty or a lack of evidence about safety and effectiveness. This is reflected in another popular aphorism in medicine Primum non nocere (First, do no harm). And just as the benefits of a therapy remain unknown when there is no strong scientific evidence, only anecdote and uncontrolled observation, so the safety is uncertain in the absence of good-quality evidence. There needs to be a very urgent need to act, and very clear disclosure of the uncertainty to clients and patients, before we use a therapy when we can’t know the true safety or efficacy.

The necessity to avoid making things worse is a major reason why we generally avoid using therapies without good evidence for their effects. It is widely accepted that if a pharmaceutical company invents a new drug, they don’t start selling it to patients on the basis that it hasn’t yet been proven not to work! These companies are required to go to great lengths to identify the risks and benefits before we are willing to give new medicines to our patients.

A low prior probability makes such a precautionary approach even more appropriate. Even if a therapy is “unproven,” in the sense of there not being much reliable research evaluating it, if the theories and assumptions behind the therapy contradict established scientific understanding, then the burden of proof is even higher, and the principle of avoiding such therapies in order not to unintentionally do harm is even more appropriate.

As an illustration, here are a few “unproven” practices that, nevertheless, most of us would follow despite the lack of controlled scientific evidence because they have a high prior probability of benefitting us:

1. Wearing a parachute when jumping out of an airplane in flight
2. Looking both ways before crossing the street
3. Taking an ambulance rather than a taxi to the hospital after having been shot in the chest

There are no high-quality clinical trials to show that these practices reduce injury or death, but it is rational to follow them anyway because of high prior probability, because they are based on well-established principles and sound reasoning.

The opposite is also true. Here are a few examples of untested and unproven practices that we would avoid despite the absence of controlled scientific evidence because the prior probability of their efficacy is very low:

1. Using one’s Qi or spiritual energy to fly when jumping out of an airplane without a parachute
2. Using The Force to detect oncoming cars when crossing the street without looking
3. Calling a cab to the hospital and waiting by the curb for it to come after having been shot in the chest

The quantity and reliability of the evidence is, of course, often more complex and in greater dispute in medicine than in examples such as these, but my point is that sometimes the absence of evidence should be taken as a reason to avoid a therapy when it has low prior probability of being safe and effective. This is a rational, common practice that we all follow in many other situations, and it makes sense in medicine as well.

Finally, it is important to remember that absolute, 100% proof is never a product of science. Even the strongest evidence can be undone by the discovery of new facts, by unidentified weaknesses in the research, or by rare events. People do survive falling out of airplanes without parachutes.

Proponents of implausible or unproven therapies often trumpet this concept as a way of defending their practices. Since nothing is absolutely certain in science, and since a few crazy ideas have proven to be true, and a few well-demonstrated claims have turned out to be false, it is tempting to conclude that one can believe anything one likes since no one knows for certain. This is pretty obviously a silly and dangerous conclusion. While people may rarely survive jumping out of airplanes without parachutes, that doesn’t make choosing not to use a parachute a sensible decision.

The corollary of the fact that science never provides absolute, eternal truth is that at some point we have to accept the evidence as “good enough.” The precise point will be fuzzy and subject to debate, but in the real world we have to be able to make decisions based on some reasonable level of probability informed by science. If a therapy has been studied extensively over a significant period of time and no evidence of a benefit has emerged, it is not rational to say forever that “no conclusion can be drawn” and “more evidence is needed.” At some point, enough is enough.

The failure to produce good evidence of a meaningful benefit despite reasonable effort is itself evidence that there is no such benefit to find. When a therapy has a low prior probability and fails to be validated after a fair effort, this absence of evidence is evidence of absence of any real benefit. With limited time and resources, we cannot afford to forever keep trying to find evidence for implausible claims that have failed multiple attempts at validation, and we harm our patients by wasting our efforts and resources in such endless and almost certainly futile efforts. The fact that we can never say with 100% certainty that a claim is false does not justify never making the pragmatic decision to ignore that claim and move on to more promising hypotheses.

It is often true that the absence of evidence is not evidence of absence. It is also sometimes true that the absence of evidence is sometimes suggestive that a claim is false, particularly when highly motivated individuals have tried to find or produce positive evidence and have repeatedly failed. And it is always true that we must make decisions in the context of some uncertainty. EBM helps us quantify this uncertainty and integrate it into our decision-making, but it doesn’t obviate making decisions. And sometimes, the most appropriate decision for our patients is that “unproven” means “risky” and, in some cases, “unlikely to be true.”

The core principle of evidence-based medicine is that not all evidence is created equal. Controlled scientific studies provide more reliable evidence than anecdotes because they contain controls for universal sources of error in human perception and judgment. This is a major feature that distinguished science from pseudoscience, real science-based medicine from faith-based practices like homeopathy and energy medicine, in which belief and “seeing with your own eyes” always trumps controlled research evidence.

The fact that controlled research is better than haphazard observation does not, of course, mean that observation is always wrong. Nor does it mean that the results every scientific study are true. In fact, when proponents of alternative therapies try to claim that their practices actually are scientifically validated, they often parade lots of scientific studies that seem to support these claims. When the flaws in these studies that undermine their conclusions are pointed out by skeptics, this is often seen as cheating, as closed-minded rejection of evidence in favor of CAM. However, critical appraisal, the evaluation of the quality and limitations of scientific studies, is another core principle of evidence-based medicine, and it applied to all research, regardless of the kind of hypothesis under study. Scientists, unlike proponents of CAM, are generally their own toughest critics, because rigorous criticism is necessary to weed out error and eventually uncover the truth about nature.

One of the leading scientists working to identify the weakness in published scientific reports, and to develop strategies for correcting them, is John Ioannidis at Stanford University.

Ioannidis, J. Why most published research findings are false. PLoS Med 2(8): e124. doi:10.1371/journal.pmed.0020124.

Despite the inflammatory title of the article, it is actually a tightly reasoned and mathematically rigorous look at specific sources of error in the published scientific literature. Knowing the features that suggest the results of a particular article may not be reliable helps one determine the degree of confidence it is appropriate to have in a particular conclusion. Here are the features Ioannidis identifies as most significant:

1. The smaller the studies conducted in a scientific field, the less likely the research findings are to be true.

2. The smaller the effect sizes…

3. The greater the number and the lesser the selection of tested relationships…

4. The greater the flexibility in designs, definitions, outcomes, and analytical modes…

5. The greater the financial and other interests and prejudices…

6. The hotter a scientific field (with more scientific teams involved)…

Many of these variables are well-known to be associated with false positive results. And while they reduce our confidence in the literature of many fields, generally the CAM literature is weaker in terms of most of these factors than the literature of mainstream scientific medicine. In particular, CAM studies are seldom replicated, usually involve small numbers of patients, often show very small, marginally significant effects, and involve considerable preconceptions or bias on the part of investigators which the studies do little to control.

Many of these factors all hinge on the degree of individual judgment or flexibility allowed in a study. The more choices the investigators have to make, in the design, conduct, and analysis of a research study, the more likely their own biases are to influence the results.  Another article from a few years ago specifically addresses this issue:

Simmons JP, Nelson LD, Simonsohn U. False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol Sci. 2011 Nov;22(11):1359-66. doi: 10.1177/0956797611417632. Epub 2011 Oct 17.

The conclusions of this paper reinforce that of Dr. Ioannidis and also emphasize a point I make here frequently; that when bias and error to creep into scientific research, they are far more likely to create the false impression of a positive result than a negative one. In other words, scientists, like all human beings, see what they want and expect to see, and if a scientific study allows these desires and expectations to influence the results, these results will tend to confirm the investigators’ beliefs even when they are false.

…flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not.

This is, as always, true in all areas, but it is a particular problem in CAM research where investigators almost always begin with strong, nearly unshakeable faith in their beliefs and where research often has poor controls for bias generally.

The point, of course, is not to suggest that scientific studies are useless and we should all go back to believing whatever we like based on our own experiences or anecdotes told to us by others. The point is that understanding the nature and severity of the weaknesses in scientific research gives us power; power to avoid excessive confidence in our conclusions and power to correct the weaknesses that reduce this confidence.

Both Dr. Ionnidis and the authors of the article in Psychological Science offer concrete measures for improving the reliability of published research. Simmons and colleagues offer these suggestions:

These measures mostly involve more transparency in the reporting of how studies are conducted. This should both help us identify weaknesses that might reduce confidence in the results and also encourage authors to address these in the design and conduct of the trial, since they know they will have to disclose them later.

In a recent paper, Dr. Ioannidis also offers some advice for improving the quality of published scientific research:

Ioannidis JPA (2014) How to Make More Published Research True. PLoS Med 11(10): e1001747. doi:10.1371/journal.pmed.1001747

His suggestions are wide-ranging, from improved statistical practices to changing the financial and career incentives for scientists. These address both the way in which personal bias influences results and some of the sources of that bias.

Box 1. Some Research Practices that May Help Increase the Proportion of True Research Findings

• Large-scale collaborative research

• Adoption of replication culture

• Registration (of studies, protocols, analysis codes, datasets, raw data, and results)

• Sharing (of data, protocols, materials, software, and other tools)

• Reproducibility practices

• Containment of conflicted sponsors and authors

• More appropriate statistical methods

• Standardization of definitions and analyses

• More stringent thresholds for claiming discoveries or ‘‘successes’’

• Improvement of study design standards

• Improvements in peer review, reporting, and dissemination of research

• Better training of scientific workforce in methods and statistical literacy

Anything human beings do is imperfect, and this applies to science as much as anything. Often, when I suggest that science is a more reliable guide to how nature works, and what is or is not safe and effective medicine, people object that science conflicts with their personal beliefs or experiences, and they trust their gut or their eyes more than controlled data. But these sources of information contain very little in the way of controls for human error. The evidence of history, and the clear improvement in our health and well-being since we began to apply scientific methods, demonstrate that science is dramatically superior as a means for gathering knowledge than such methods.

It is true, however, that even the best methods we have for obtaining knowledge are still imperfect and still involve some freedom for bias to enter into our conclusions. Relying on science does not mean blinding trusting the results of every single study. Evidence-based medicine requires that we carefully and critically evaluate individual studies and the process of scientific research as a whole, always seeking to identify and reduce error. Merely criticizing or dismissing science as imperfect is not, in itself, useful. Such criticism must be sufficiently specific and focused to allow for strategies of improvement, and must contain at least an implicit recognition that science is still the best tool we have for understanding how nature works.

Human beings are diverse. We are diverse, as individuals and as communities, in our physical characteristics and in our experiences. This leads to a diversity of beliefs and opinions. It is inevitable that we will disagree with other people about many things, large and small. And it is also inevitable that these disagreements will sometimes engender strong, often primitive and unpleasant emotions. However, it is not inevitable, and it should never be accepted or acceptable, that such disagreements lead to violence.

In his book, The Better Angels of Our Nature, Stephen Pinker makes a strong and compelling case that despite the awful things people do to one another daily around the world, and our even greater and more immediate exposure to these acts through the media, we are actually living in the safest and most peaceful time yet in human history. This is not because our biology or our fundamental nature as human beings has changed. It is because we have exercised our inherent abilities to learn, to mature, and to set limits to our own behavior. The result is often called “civilization,” and it represents a realization of at least some of the human potential to manage our feelings and our disagreements in a way that limits the damage they do and allows a measure of peace and freedom to as many people as possible.

Yesterday in Paris, there was a small, horrible, and very clear attack against civilization. Whatever the details of your beliefs, about religion or any other subject, if you have ever had a belief or opinion that you wanted to share, if you have ever disagreed with someone else’s beliefs and felt the need to say so, if you believe there is or can be such a thing as civilization, then you should feel personally attacked and personally threatened by the murder of journalists, cartoonists, police officers, and bystanders at the offices of Charlie Hebdo.

Much of the content of this blog involves criticism and critique of the claims and beliefs of others. I try to make my criticism substantive and respectful and to focus on challenging ideas and claims about the world, not attacking individuals. I do this because I believe it is necessary to give animal owners the facts and information they need to make good healthcare choices for their pets and because I feel an obligation to make a contribution to my profession and the society I live in.  While I do not wish to hurt or offend people, I understand that such critiques will inevitably be taken personally and cause offense. This, again, is in the nature of human disagreements, and it is unavoidable if we are to strive for progress and to separate good ideas from bad ideas through debate and discussion.

As a result of my writing, I have certainly received plenty of hate mail and personal verbal attacks. I have been threatened with legal action on more than one occasion in order to suppress my criticism of others. I have not yet been threatened with physical violence, though some comments have come close: “You are a lying scumbag…… eat dog shit and die !!!! you evil money monger !!!!” But in some small way, I have been involved in the attempt to sustain and contribute to civilization through civil discourse and debate, and I have been touched in a very small way by the impulses to destroy this aspect of civilization that led to the atrocity in Paris. I’m not, in any sense, as brave or important to this process as the people who lost their lives at Charlie Hebdo, but I feel a kinship and a deep respect for their commitment to civilization, which has cost them so much.

I do not read French and have not been a follower of Charlie Hebdo. From what I’ve seen in the last day, I suspect I wouldn’t like or agree with much of the satire for which the people at this magazine were attacked. But that does not lesson my belief that they had the right to produce and publish that satire, and that the attack on them is an attack on anyone who values individual freedom, freedom of belief and expression, or anything that deserves to be called civilization.

As has been said more eloquently by others before, people deserve dignity and respect, ideas do not. Even ideas people feel strongly about, such as religion, must be subject to critical evaluation, even satire, or the kind of growth and progress of civilization that has made life better for all human beings is impossible. People of all faiths and none have appropriately condemned the physical attack in Paris and the symbolic attack against freedom of thought and expression it represents.

I can think of no more appropriate way to reject this kind of barbarism and intimidation than to reproduce some of the cartoons that these attackers felt justified this kind of violence, as well as some of the responses that have been published since. If you are offended, I am sorry. But I would ask you to remember that your beliefs and opinions are just as offensive to someone else somewhere. For any one of us to have true freedom, we must defend freedom for all of us. Je suis Charlie. Nous somme tous Charlie.