The SkeptVet Blog

CAM proponents are very fond of citing the antiquity and lineage, fictional or not, for their methods. Acupuncture, Traditional Chinese Medicine, and many herbal nostrums are promoted as “time-honored,” with the implication or explicit statement that anything people have been doing for so long must be a good thing. Foolishly expecting reason and logic to have some impact on this, we skeptics often try to point out that the number of people who believe something and the duration of time they believe it is not, unfortunately, a reliable indicator of whether or not it is BS. Or, as Tim Minchin more eloquently puts it in one of his songs, “I don’t believe that just because ideas are tenacious it means that they’re worthy.”

Soldiering on despite the blank stares or scoffing such a response often engenders, we continue to offer facts as rebuttal. The dramatic increase in life expectancy and health as scientific medicine came to replace traditional folk medicine seems a particularly salient fact in this context, but I recently ran across another smaller but more specific example of medical progress through time.

Dr. Atul Gawande, of Complications fame wrote an article for the New England Journal of Medicine in 2004 entitled Casualties of War–Military care for the wounded from Iraq and Afghanistan. As part of his discussion of care for wounded soldiers, he presented some historical statistics on the lethality of wounds suffered in combat by U.S. soldiers since the Revolutionary War. The table below illustrates the findings:

While we don’t have numbers from earlier eras or other countries, and while some caution is indicated in looking at numbers as old as those in the 18th and 19th centuries, the pattern is quite interesting. Survival varied, likely due in part to the reliability of the statistics, especially for small conflicts such as the Spanish-American War, but was consistently at or below 30% for most of the period until the 1950s. It improved to a consistent 25% for conflicts from then until the Iraq and Afghanistan conflicts and then improved dramatically. 90% of wounded soldiers can be expected to survive and come home, and while this raises legitimate questions about  how we ensure they have an acceptable quality of life and adequate help dealing with the lasting effects of their injuries, very few of them over the long term will likely prefer to have died. In this limited sense, such a change represents true progress in medical care for these servicepeople.

Dr. Gawande doesn’t address the changes prior to the current wars, but it seems likely that the development of antibiotics, blood replacement products, and other crucial developments in trauma care are responsible for the improvement after WW II. Gawande does delve into the details of the changes in medical practices that have dramatically improved survival in the current conflicts.

The point from the perspective of the difference between traditional and science-based medicine is that 1) the improvement coincides with major developments in scientific medical care which are often dismissed as irrelevant or even sources of harm and 2) medical practices are constantly being re-evaluated and improved based on measurable results. This is in stark contrast to the claims of those who argue for faith in tradition and received ancient wisdom.

Traditional folk medical practices have indeed often been around for centuries or even millennia, though these claims are sometimes exaggerated or fabricated by CAM advocates. However, in all those centuries or millennia, these methods have never succeeded in making the improvements in survival and health that have come in the last two hundred years, as Enlightenment rationality and scientific medicine have been born and matured. And these traditional practices are founded on the perceived virtue of not changing, whereas the ability to change and adapt that science gives mainstream medicine is one of its greatest strengths. This paper provides a small but clear illustration of the general case that the virtues of the scientific method demonstrably provide better results than the claimed virtues of tradition and longevity often attached to alternative medical practices.

One of the most popular nutritional supplements these days is fish oil. It contains a high proportion of omega 3 fatty acids (EFA), notable eicosapentaenoic acid (EPA) and docosahexaenoic acid (DHA). This supplement is purported to have a broad range of beneficial effects in many disease conditions due to its effect on chemicals in the body involved in the inflammatory response (for more details see this article on eicosanoids). Some of these effects, such as a reduction in the rate of heart attacks in people with established heart disease and heart attack risk factors, are well-supported by research data. Other claims are less clearly valid. Several literature summaries are available from the Agency for Healthcare Research and Quality, and Medline.

In the veterinary arena, EFA supplements are widely used for allergic skin disease, with modest supportive clinical trial evidence (see also 1, 2, 3, 4).  In humans, there is limited evidence to support an effect on some clinical variables in patients with rheumatoid arthritis, and some have suggested osteoarthritis treatment as a veterinary application for these supplements. There is reasonable biologic plausibility to support investigating this use of EFA supplements, and two articles in a recent issue of the Journal of the American Veterinary Medical Association (JAVMA) report studies evaluating the use of fish oil supplements for dogs with confirmed osteoarthritis. I shall briefly review each of them.

The first study [1] was a multicenter, randomized, double-blind, placebo-controlled prospective study lasting 24 weeks. 167 dogs were randomized into two groups, one receiving a diet supplemented with EPA and DHA, the other receiving a pretty closely-matched control diet. There were no significant differences in any relevant variables between the groups at baseline. 23% of the dogs failed to complete the study (9% in the treatment group and 14% in the control group), and there were no significant differences between the groups with respect to these dropouts.

Assessment measures were subjective, with an owner survey and a clinical assessment by participating veterinarians. Dogs were evaluated at 6, 12, and 24 weeks after beginning the diets. Bloodwork showed significant increases in the blood levels of EHA and DHA in the dogs fed the test diet, so these substances were clearly absorbed. The owner survey evaluated 13 measures of comfort and function over three time periods (0-6 weeks, 6-12 weeks, and 12-24 weeks on the diets). Of these 2 measures were significantly different between the groups at the first evaluation, and 1 measure was different at the last evaluation. There were no significant differences between the groups in clinical evaluation by veterinarians.

The study was apparently well-designed and well-conducted. The measures of outcome were subjective, which is less than ideal. It seems fairly clear that the results do not support the use of EFA supplementation for osteoarthritis. After all, out of 39 possible points at which the groups were compared in terms of owner evaluation only 3 showed chances not attributable to chance, and none of the evaluations by veterinarians showed significant difference. Unfortunately, the discussion section of the article is less an objective survey of the trial or the overall preponderance of the evidence than it is an attempt to put the most positive possible spin on the results.

The authors suggest their subjective measurement instrument may not have been sufficiently sensitive to detect a change and try to attribute the failure of the veterinarians to detect a difference to the hospital environment and limitations on clinical evaluation. They then conclude, “Our results suggest an ameliorative effect of omega-3 fatty acid supplementation in arthritic dogs,” and “ingestion of the test food….appeared to improve the arthritic condition in pet dogs with osteoarthritis.” Such a conclusion so clearly  at odds with the data presented in the report seems to be more an example of confirmation bias and cognitive dissonance than a reliable presentation of the evidence.

The second study [2] was conducted by several of the same authors. It too was a well-designed randomized, double-blinded, placebo-controlled, prospective study of the effects of dietary EFA supplementation on dogs with osteoarthritis. 44 dogs were enrolled in the study, and 14% failed to complete it (9% in the treatment group and 5% in the control group). Again there were no significant differences detected at baseline and no differences with respect to dogs that failed to finish the study. Both subjective and objective measures were evaluated at baseline and after 45 and then 90 days of the diets. Owner and clinician subjective evaluations were compared as were the results of force plate gait analysis.

In contrast to the previous study, there were no significant differences in owner evaluations of the subjects’ comfort and function. The authors attributed this to the low number of subjects rather than the more parsimonious explanation that there was no differences of sufficient magnitude to be noted. The clinician evaluations at 90 days showed significant differences from baseline for the test group in 3 of 5 measures. However, there were no significant differences between the test and control group for any measure.

The authors stated, correctly in my opinion, that “subjective assessment of limb function lacks repeatability as an outcome measure and is inferior to objective data obtained from force platform gait analysis.” Such an analysis was performed on all subjects. The results showed no change from baseline to 90 days for any of six variables measured (peak vertical force, vertical impulse, braking and propulsive peak forces, and braking and propulsive impulses). The test group did show a statistically significant difference in the mean percentage change in one measure, peak vertical force.

Again, these results provide lackluster support for the contention that EFA supplementation may be beneficial for dogs with osteoarthritis. Some subjective clinical measures showed a difference, but this is not consistent with the results of the other, larger trial, and the authors themselves minimized the significance of these results in both papers. One objective measure did show a statistically significant, and likely clinically significant change. However, the combined results of the two trials offer tepid support for the hypothesis under examination, and an interpretation of no meaningful effect seems much better supported by the results.

Unfortunately, the authors again spin these results in the most positive way possible:

“Together with the findings of our other study, findings of the study reported here supported the hypothesis that ingestion of fish oil omega-3 fatty acids improves clinical signs in dogs with osteoarthritis….Dietary supplementation with fish oil omega-3 fatty acids resulted in an improvement in weight bearing in dogs with osteoarthritis.”

The authors do acknowledge some of the limitations of their study and suggest that further research is necessary for definitive recommendations. I would agree that these results might justify further stud, though I see no reason to expect dramatic findings of benefit. However, the reality is that in the world of veterinary medicine, with limited resources and clinical trial evidence, the more likely outcome of these reports is going to be an increase in the prescription for EFA supplements intended to treat osteoarthritis. The positive statements in the abstracts and discussion sections are likely to be the “take-home” message many readers get from the reports, despite the reality that the data is considerably less positive. I always appreciate well-designed and conducted research, but these reports emphasize the difficulty in conducting such trials without having an investment in their outcome that affects the interpretation of the results. This is a large part of why careful and critical evaluation of the primary literature for oneself is such a key component to sound, evidence-based practice.

References

1. Roush JK, Dodd CE, Fritsch DA, Allen TA, Jewell DE, et al. Multicenter veterinary practice assessment of the effects of omega-3 fatty acids on osteoarthritis in dogs. J Am Vet Med Assoc. 2010 Jan 1;236(1):59-66.

2. Roush JK, Cross AR, Renberg WC, Dodd CE, Sixby KA,, et al. Evaluation of the effects of dietary supplementation with fish oil omega-3 fatty acids on weight bearing in dogs with osteoarthritis. J Am Vet Med Assoc. 2010 Jan 1;236(1):63-73.

Mark Crislip at Science-Based Medicine recently discussed Hill’s Criteria of Causation, but after looking at Dr. Hill’s original paper I felt obliged to examine the subject here as well because it is central to science-based medicine, and key in differentiating real medicine from quackery.

As I often discuss, a crucial issue in medicine is the question of epistemology; how we know what we know. Our therapies ought to be based on a real understanding of health and illness, and sound evaluation of the safety and efficacy of the interventions we undertake. The key difference between scientific and alternative medicine is in the criteria  for such evaluation. CAM proponents are often satisfied with theories of disease than are either made up out of whole cloth by individuals (e.g. homeopathy, chiropractic) or that are an agglomeration of folk beliefs (e.g. acupuncture, reiki and other energy therapies, TCM). Whether these theories are consistent with our general understanding of the universe, or with each other, is not a concern.

Even more damning, CAM practitioners are frequently satisfied that their personal impressions and experiences are adequate to validate the truth of the causal associations they see underlying disease or in the response to their treatments. Despite the manifold errors the human mind is known to exhibit in making causal associations, these people don’t see any need for a type of validation that compensates for these errors. They may lay claim to any scientific evidence that can be found for their approaches, for the aura of legitimacy it confers, but they will never reject the conclusions of their experience or intuition in light of any other kind of evidence.

In 1965, Dr. Sir Austin Bradford Hill gave a talk to the Royal society of medicine entitled The Environment and Disease: Association or Causation? In this address, he laid out nine criteria for concluding that an observed association was in fact representative of a causal relationship. His lucid and cogent presentation provides us with an excellent framework for making decisions about the causes of disease or the effects of treatments, all the more because he was so careful to emphasize that his criteria were guidelines which should support, not replace critical thought and judgment.

Here are Dr. Hill’s criteria, with a brief explication of each:

1. Strength- The strength of an association can be supportive of a true underlying causal relationship. If a proposed cause for a disease is associated with the disease itself only sporadically or unpredictably, this is weaker evidence for its causal role than if it is reliably present in conjunction with the disease. As always, Dr. Hill cautioned that “we must not be too ready to dismiss a cause and effect hypothesis merely on the grounds that the observed association appears to be slight,” but the strength of an association is a relevant factor in evaluating its significance.

2. Consistency- An association between cause and effect should be robust enough to be demonstrable in multiple studies by different investigators. This is key to understanding the fallacy of presenting a single study as definitive evidence for or against a disease etiology or treatment effect. The preponderance of the evidence as it accumulates is more meaningful than the results of particular studies, though the quality as well as quantity of the evidence must also be considered.

3. Specificity- If a putative cause is associated with a very specific set of symptoms, or a treatment with very specific effects, this supports a causal relationship. If the cause is present with a wide variety of different clinical presentations or the results following a treatment are highly variable, this argues against a causal relationship. As always, Dr. Hill correctly cautions against excessive rigidity in the application of this criterion, pointing out that many diseases are multifactorial and may not exhibit great specificity of association with a single causal factor. As he puts it, “if specificity exists we may be able to draw conclusions without hesitation; if it is not apparent, we are not thereby necessarily left sitting irresolutely on the fence.”

4. Temporality- Causes by definition precede their effects, so if a potential causal agent is observed after the condition it is speculated to be causing, this argues strongly against a true etiologic relationship. The obverse of this is, of course, the post hoc ergo propter hoc fallacy, perhaps the most intransigent and troublesome reasoning error in medicine. The observation that one thing precedes another is not in any way evidence for a causal connection because of the myriad of alternative explanations that often turn out to dictate the order of precedence. Unfortunately, precedence is very compelling as proof of causality despite being unreliable.

5. Biological Gradient- Also known as a dose-response curve, this relationship is common in biological systems and is strongly supportive of a causal/effect relationship. If the amount of a drug given or the intensity of an exposure to a potential cause of disease is correlated with the likelihood or severity of the resultant effect or the disease, this supports a conclusion of causation. If, however, a little bit is as good (or bad) as a lot, the hypothesis of causation should be questioned.

6. Plausibility- Dr. Hill was less convinced of the reliability of this criterion because, as he correctly observed, “What is biologically plausible depends upon the biological knowledge of the day.” However, as he explains in discussing his subsequent criterion, a proposed relationship which contradicts well-established understandings in biology or other areas of knowledge should be viewed as less likely to be correct, and less worthy of implementation or investigation, than a relationship consistent with such understandings.

7. Coherence- In contrast to his caution regarding the criterion of plausibility, Dr. Hill states, “On the other hand, the cause-and-effect interpretation of our data should not seriously conflict with the generally known facts of the natural history and biology of the disease.” A proposed mechanism, such as that underlying homeopathy, which requires overturning well-established principles in biology, chemistry, and physics must face a far greater burden of proof to be accepted as valid than a hypothesis consistent with established knowledge.

8. Experiment- Clearly, the results of controlled laboratory, animal model, or clinical trial studies are critical in drawing firm conclusions about the causation of disease or the effects of medical interventions. Contrary to the stereotype sometimes promulgated by CAM advocates, scientific medicine does not rely solely on this criterion, but it can be a very powerful tool for confirming or invalidating proposed relationships.

9. Analogy- This is perhaps the weakest of Hill’s criteria, though it is popular because of the simplicity and apparent clarity of arguments by analogy. While toxins may cause illness, for example, not all illness should be attributed to toxins. Likewise, while infection may cause a fever, not all fevers are due to infection. Analogies are most useful for suggesting possible relationships, which should then be confirmed or disproven by application of more rigorous criteria.

After discussing his criteria, Dr. Hill goes on to eloquently addresses two critical issues in the epistemology of medicine. First, he is very clear that conclusions about cause and effect relationships must be based on assessment of multiple criteria and application of sound reasoning, not on slavish adherence to one criterion or to any algorithmic model, including his:

“What I do not believe – and this has been suggested – that we can usefully lay down some hard-and-fast rules of evidence that must be obeyed before we can accept cause and effect. None of my nine viewpoints can bring indisputable evidence for or against the cause-and-effect hypothesis and none can be required as a sine qua non. What they can do, with greater or less strength, is to help us to make up our minds on the fundamental question – is there any other way of explaining the set of facts before us, is there any other answer equally, or more, likely than cause and effect?”

This shows a sophistication of reasoning inconsistent with the stereotype of science-based medicine advocates as obsessed with randomized clinical trials and blind to all other evidence for or against proposed causal relationships.

Dr. Hill also presciently addresses the proper role of statistical analysis in medical decision making. In dealing with questions of cause and effect, he argues that “no formal test of significance can answer those questions. Such tests can, and should, remind us of the effects that the play of chance can create, and they will instruct us in the likely magnitude of those effects. Beyond that they contribute nothing to the “proof” of our hypothesis.” He goes on to discuss how statistical analysis entered medical research in the 1920s and 1930s as a needed anodyne to the practice of basing firm conclusions and therapeutic practices on small, not representative case examples. However, he speaks strongly of the excessive role such methods were then coming to play in drawing conclusions from medical research, and the problems he describes have only worsened in the intervening time:

“I am told, some editors of journals will return an article because tests of significance have not been applied. Yet there are innumerable situations in which they are totally unnecessary – because the difference is grotesquely obvious, because it is negligible, or because, whether it be formally significant or not, it is too small to be of any practical importance. What is worse the glitter of the t table diverts attention from the inadequacies of the fare….Of course I exaggerate. Yet too often I suspect we waste a deal of time, we grasp the shadow and lose the substance, we weaken our capacity to interpret the data and to take reasonable decisions whatever the value of P. And far too often we deduce ‘no difference’ from ‘no significant difference.’ Like fire, the chi-squared test is an excellent servant and a bad master.”

Lastly, Dr. Hill answers definitively and in no uncertain terms those critics who falsely accuse practitioners of science-based medicine of passivity or helplessness in the absence of definitive clinical trial evidence:

“Finally, in passing from association to causation I believe in ‘real life’….In asking for very strong evidence I would, however, repeat emphatically that this does not imply crossing every ‘t’, and swords with every critic, before we act.

All scientific work is incomplete – whether it be observational or experimental. All scientific work is liable to be upset or modified by advancing knowledge. That does not confer upon us a freedom to ignore the knowledge we already have, or to postpone the action that it appears to demand at a given time.

Who knows, asked Robert Browning, but the world may end tonight? True, but on available evidence most of us make ready to commute on 8:30 the next day.”