Though I’m a little late, I’m a forgiving sort, so I won’t be mad at myself for forgetting my blogiversary. June 4 marked five years of the SkeptVet Blog! The project has grown in so many ways. In the early days, I was excited to get fifty hits in a day, and in the last year it has been routine to get 800-1000 per day. Still small potatoes by Internet standards, but gratifying nonetheless. My goal has always been to provide a useful informational resource to pet owners and veterinarians, and to provide an alternative perspective on many therapies for which the only information on the internet is biased and unjustified marketing hype. The more people who know the blog is here, the more useful it is likely to be.

I have also expanded to discuss a much wider range of topics, even leaving the domain of alternative medicine more and more often. And in writing articles every week for five years, hopefully I have become a more effective communicator. I cringe when I look at the style and tone of some of my early posts, but I choose to view that as a good sign since it must mean I have grown and changed.

Writing this blog has provided me with opportunities to talk about evidence-based medicine more widely, to veterinarians and students and the general public, through my own lectures and presentations, writing opportunities, and interviews in the media. Hopefully, these activities will add an under-represented perspective, that of science and skepticism, to conversations within these communities about the best way to protect and restore health in our animal companions. I’ve never claimed to have all the answers, but I think I have some important and useful questions.

And writing this blog has certainly stimulated discussion and feedback. I have reviewed the negative comments before because I believe they are instructive in understanding the beliefs, values, and emotions that support pseudoscience and impede the effective use of science to improve veterinary medicine and animal health. But as an anniversary present to myself and those who find this project useful, I thought I’d review some of the positive feedback I’ve received. As well as being fun, this will help maintain the energy needed to keep going, which I fully intend to do. (All comments have been anonymized).

As a side note, another way to maintain this energy is going on vacation, which I will be doing starting this weekend. So activity, including responses to comments and emails, will be pretty light here for a few weeks. Never fear, I will keep up as best I can while I’m travelling, and I will dive back in with responses to your questions and comments and new articles with renewed vigor when I get back.

Finally, I offer a heartfelt “Thank you!” to all who have taken the time to provide supportive comments, questions or suggestions for topics to discuss, and thoughtful, substantive criticism. Your feedback does a great deal to neutralize the vacuous and angry responses a blog such as this inevitably draws and to maintain my faith in the possibility of civil, rational, fact-based discourse.

Thank you very much for taking the time to respond to my email and for providing a thoughtful response. .. Thank you for all the information you provide on your website as well.

Thank you. I asked my vet about XX again after I read your post…. I want to thank you for your honest “second” opinion before I started her on both meds. Knowing the possibility of “unknown” side effects makes me want to rethink…

Thank you very much for the information. .. Again, your advice is much appreciated in a difficult time for us.

I was very pleased the day I found your blog. You have excellent epidemiologic insight, which is rare among vets…. So, I applaud you for your post on media coverage of healthcare research.

Thank you so much for your timely response and thoughtful answers. I greatly appreciate you effort.

Thank you so much for your reply and the referenced article…Thank you so much for taking the time to answer my original email. I never expected to receive a reply, let alone a quick reply. The ACVIM Consensus Statement was extremely helpful.

I am so touched that you have replied so promptly and it is really fantastic to have such a well considered and scientifically based answer. Thank you very much, I truly appreciate it.

Thank you so much for your quick and detailed reply! I will be reviewing the links today. I so appreciate the work that you do and your personal reply to inquiries. I know that you receive a lot of negative feedback but I know there are also many, including myself, that appreciate your time, research, analysis and pragmatic consideration.   As a pet owner who loves her dog, I of course am in search of anything that won’t harm but could be helpful beyond the standard of care that we will be doing. I try to do diligent research on any recommended unconventional treatments.  Not just for my own dogs but when I am assisting in the treatment of patients at our hospital…This is something I take very seriously.  Reviewing your information I hope will help me make the best, most informed decision…

I just wanted to thank you for your blog. I have training in research design and statistical methods, but not as a biologist or veterinarian. I always want to read peer-reviewed research when something is wrong with my dog, but don’t understand the medical terms. Your blog is really a godsend. Keep up the good work!

I had originally intended to reply…in a private e-mail with encouragement and praise for speaking up for the rest of us in the “silent majority.”  Its a tough role to play and I continue to be impressed with your eloquence…Keep it up.

I’m so glad to see some common sense in the world. Your blog exposes a lot of the craziness which I really appreciate, especially coming from an MD background… It’s fascinating to me how quickly people gravitate towards alternative medicine, towards theories that sound magical and assume that they are just correct without any scientific evidence to back it up. Thanks for keeping it real!

I did try do my own review from primary literature, but I do not know the journals or the field at all, so your review was exactly what I was looking for, thank for that! My local vet now has a few extra facts to consider…As for the rest of your site, I enjoyed that too, and I’m very thankful that you take strong, and informed standpoints on CAM issues.

Good luck, and keep up the good work!

I wanted to drop a quick note to thank you for taking the time to post your thoughts and observations regarding various common and/or popular veterinary treatments…The net is long on anecdotes, short on science. Finding your blog has gone a long way towards filling the gap.

Just a quick thank you for your website and blog site. I’ve followed your sites for a while, and your citations to sources and reference materials are helpful in allowing me to share your information to family, friends, and our vets. The (sometimes) desperate urge to “do something” for our ailing pets clouds the minds of owners and vets alike, it seems. And some have a very hard time getting out the cloud.

First I wanted to thank you for posting your lecture, I shared it far and wide and hope that some of my colleges and friends were able to get what I was able to get out of it…Thank you for your response, your response far exceeded my expectations and I appreciate that.

You are doing a great service to pet companions of all stripes! we just struggled with putting down our beloved dogs in the past year, and the pressure to do all sorts of extraordinary but probably not effective treatments for them made it much more stressful.

Thank you so much for taking the time to reply!  Your message has helped very, very much.  We (pet people and their pets) are fortunate to have someone as thoughtful and generous as you are.

I’m new to your website as of today, came across it while searching for evidence-based information on dog food. What a great find! I completely appreciate your world view…Thank you for providing such a great source of well-researched and thought out information.

Just a quick note to tell you of my appreciation for you site. For years I have tried to educate my clients about the problematic proof supporting many of the alternative therapies other clinicians have been promoting… keep up the good work…

I appreciate your articles and perspectives. It’s so difficult to know how to best care for one’s pets in the face of so much contradictory information out there. .. Thanks so much!

Want to thank you for all of your efforts. Greatly appreciated and just hope that the folks who should be reading your site, somehow find it.

Thank you so much for your response. It is quite disheartening that there are so few veterinarians who are willing to speak out against organisations such as the American Council of Animal Naturopathy.

Skeptvet has superpowers It takes not only a extensive knowledge of vet medicine but the ability and time to put in writing that knowledge so others can understand the issues. I have seen other vets with extensive general small animal and human medicine knowledge. I have seen other vets that were great writers. But never a vet that had both knowledge with his ability to write. I find it difficult to believe only one person is at work here in this blog. It’s amazing what one person in the field of vet medicine is capable of accomplishing if they have the knowledge, training and dedication to do it.

Thanks for keeping it real, skeptvet, it’s frustrating to read some of the rants that occur here by those with an obvious ax to grind.

Skepvet, I’ve enjoyed the dialogue between yourself and dfg. I also appreciate your blog as source of information.

Thanks for the science-based common sense!

Thank you, Thank you !!! For writing such a clear, well documented and intelligent and thorough response and work.

Thanks for taking the time to compose this article…Your review was applicable and timely.

Hiya, just wanted to say and thanks. I’m getting a kitten and was considering a barf diet, but after doing my research, and coming across your site, I’m relieved to say I won’t be feeding my pedigree any raw food.

I just found this blog, and I’m so glad I did. There is way too much woo and pseudo-science out there for people who genuinely care about companion animals to wade through when trying to make informed choices on anything from medication and nutrition to training techniques.

Caveat emptor. Skeptvet, thank you for taking the time to elaborate. I found it helpful and I hope that others will as well.

Thank you for providing this valuable perspective.

I am still looking for the product I need. But, in the meanwhile I read your blog, loved it. Made perfect sense to me. And such a perfect way to confuse the writers, with fact.

Thanks for what you do generally, and specifically for providing a clear and reasonable overview of the literature on this issue. Like you and many others, I find the lack of quality controlled research here unfortunate

I am SO happy to have found your website! Will be following faithfully form now on. Thanks!!

Excellent blog post, as always, Skeptvet. Thank you for all you do.

Thank you again for producing this blog. I’m sharing this entry as part of my ethical duty to spread knowledge about the inefficacy of homeopathy.

Thanks for yet another well presented summary of the fallacies of magical thinking,

Thank you so much for this excellent summary based on SCIENCE.

Someone needs to stand up to faith masquerading as science, and people who would try to twist evidence to support their own agenda. Someone needs to hold the high ground when such people start flinging ad hominem abuse and sarcastic crap in defence of their idealism. But it’s so hard, and most of the time it’s a thankless task. I feel for you.

Thank you so much for offering a balanced view point of this.

Thanks for the facts. I trust a well argued appeal to reason and peer-reviewed evidence over the numerous emotional online sources decrying the substance.

Thank you thank you thank you ! The Dr. Dean Edell of veterinary medicine

Earlier this year, I reviewed a research study claiming that dogs orient themselves to fluctuations in the earth’s magnetic field when defecating. I was asked to re-post the review on Publons, a site which publishes reviews of journal articles. The authors have posted a response there which answers some concerns I expressed in my review, but which also illustrates some of the same misconceptions about statistics and hypothesis testing that I originally discussed. I have responded both here and on the Publons forum.

The summary of the reactions of the media on our paper is very fitting and we agree. The critic of our study is, however, biased and indicates that the author did not read the paper carefully, misinterpreted it in some cases, and, in any case is so “blinded” by statistics that he forgets biology. Statistics is just a helpful mean to prove or disprove observed phenomena. The problem is that statistics can “prove” phenomena and relations which actually do not exist, but it can also “disprove” phenomena which objectively exist. So, not only approaches which ignore proper statistics might be wrong but also uncritical sticking on statistical purity and ignoring real life.

To begin with, I believe the author and I agree that statistics are easily and commonly misused in science. Unfortunately, this response seems to perpetuate some of the misconceptions about the role of statistics in testing hypotheses I discussed in my original critique.

Statistics never prove or disprove anything. Schema such as Hill’s Criteria of Causation and other mechanisms for evaluating the evidence for relationships observed in research studies illustrate the fact that establishing the reality of hypothesized phenomena in nature is a complex business that must rest on a comprehensive evaluation of many different kinds of evidence. It is unfortunate that p-values have become the sine qua non of validating explanations of natural phenomena, at least in medicine (which is the domain I am most familiar with). The work of John Ionnidis and the growing interest in Bayesian statistical methods are examples of the move in medical research to address the problem of improper use and reliance on frequentist statistical methods.

That said, these methods do have an important role in data analysis, and they contribute significantly to our ability to control for chance and other sources of error in research. The proper role of statistical hypothesis testing is to help assess the likelihood that our findings might be due to chance or confounding variables, which humans are notoriously terrible at recognizing. If we employ these tools improperly, then they cease to fulfill this function and instead they generate a false impression of truth or reliability for results that may easily be artifacts of chance or bias.

The authors accuse me of being “so ‘blinded’ by statistics that he forgets biology.” This is ironic since their paper uses statistics to “prove” something which a broader consideration of biology, evolution, and other information would suggest is improbable. Even if the statistical methods were perfectly and properly applied, they would not be “proof” of anything any more than improper use of statistics would be definitive “disproof” or the authors’ hypothesis. While I discussed some concerns about how statistics were used in the paper, my objections were broader than that, which the authors do not appear to acknowledge.

The author of this critic blames us of “data mining”. Well, first we should realize that there is nothing wrong about data mining. This is an approach normally used in current biology and a source of many interesting and important findings. We would like to point out that we have not “played” with statistics in order to find out eventually some “positive” results. And we have definitively not sorted data out. We just tested several hypotheses and always when we rejected one, we returned all the cards (i.e. data) into the game and tested, independently, anew, another hypothesis.

Though I am not a statistician, I believe there is a consensus that while exploratory analysis of data is, of course, appropriate and necessary, the post-hoc application of statistical significance tests to data after patterns in the data have already been observed is incorrect and misleading. This is what the paper appeared to suggest was done, and this would fit the definition of inappropriate data-dredging.

Note also that we performed this search for the best explanation in a single data sample of one dog only, the borzoi Diadem, for which we had most data. When we had found a clue, we tested this final hypothesis in other dogs, now without Diadem.

This was not indicated in the description of the methods provided in the original paper. If the exploratory analysis was done with one data set while the authors remained blind to the data set actually analyzed in the paper, then that would be an appropriate method of data analysis. The subsequent statistically significant results would not, of course, necessarily prove the hypothesis to be true, but they would at least reliably indicate the likelihood that they were due solely to chance effects.

This does not, however, entirely answer the concern that the study began without a defined hypothesis and examined a broad range of behaviors and magnetic variables in order to identify a pattern or relationship. As exploratory, descriptive work this is, of course, completely appropriate. But the authors then use statistical hypothesis testing to support very strong claims to have “proven” a hypothesis not even identified until after the data collection was completed. This seems a questionable way to employ frequentist statistical methods.

Let us illustrate our above arguments about statistics and “real life” on two examples. Most medical diagnoses are done through exclusion or verification of different hypotheses in subsequent steps. Does it mean that when the physician eventually finds that a patient suffers under certain illness, the diagnosis must be considered improbable because the physician has already before tested (and rejected) several other hypotheses?

This analogy is inapplicable. The process of inductive reasoning a clinician engages in to seek a diagnosis in an individual patient is not truly analogous to the process of collecting data and then evaluating it statistically to assess the likelihood that patterns seen in the data are due to chance. Making multiple statistical comparisons, particularly after one has already sought for patterns in the data, invalidates the application of statistical hypothesis testing. The fact that in other contexts, and without the use of such statistical methods, people consider possible explanations and then accept or reject them based on their observations is irrelevant.

Or imagine that we want to test the hypothesis that the healthy human can run one kilometer with an average speed of 3 m/s. We find volunteers all over the country who should organize races and measure the speed. We shall get a huge sample of data, we have an impression that our hypothesis is correct but the large scatter makes the result insignificant. So we try to find out what could be the factors influencing speed. We test the age – and find out that indeed older people are slower than younger ones, so we divide the sample into age categories, but the scatter is still too high, so we test the effect of sex, we find a slight influence, but it still cannot explain the scatter, we test the position of the sun and time of the day, but find no effect, we test the effect of wind, but the wind was weak or it was windless during races, so we find no effect. We are desperate and we visit the places where the races took place – and we find the clue: some races were done downhill (and people ran much faster), some uphill (and people ran much slower), those who ran in flat land ran on average with the speed we expected. So we can now conclude that our hypothesis was correct and moreover we found an effect of the slope on running speed. We publish a paper describing these findings and then you publish a critic arguing that our approach was just data mining and was wrong and hence our observation is worthless and that the slope has no effect on running speed at all. Absurd!

Again, this example simply describes a process for considering and evaluating multiple variables in order to explain an observed outcome, which is not the objection raised to the original paper. If the only hypothesis in a study such as described here was that at least one human being could run this fast, then a single data point would be sufficient proof and statistics would be unnecessary. However, if one is trying to explain differences in the average speed of different groups of people based on the sorts of variables mentioned, the reliability of the conclusions and the appropriateness of the statistical methods used would depend on how the data was collected and analyzed. In any case, nothing about this has any direct relevance to whether or not the data collection and analysis in the original paper was appropriate or justified the authors’ conclusions.

As I said in the original critique, this study raises an interesting possibility; that dogs may adjust their behavior to features of the magnetic field of the earth. The study was clearly a broadly targeted exploration of behavior and various features of the magnetic environment: “we monitored spontaneous alignment in dogs during diverse activities (resting, feeding and excreting) and eventually focused on excreting (defecation and urination incl. marking) as this activity appeared to be most promising with regard to obtaining large sets of data independent of time and space, and at the same time it seems to be least prone to be affected by the surroundings.” It did not apparently start with a specific, clearly defined hypothesis and prediction, so in this sense it seems an interesting exploratory project.

However, with such a broad focus, with mostly post-hoc hypothesis generation, and with a lack of clear controls for a number of possible alternative explanations, the study cannot be viewed as definitive “proof” of the validity of the explanation the authors provide for their observations, though this is what is claimed in the paper: “…for the first time that (a) magnetic sensitivity was proved in dogs, (b) a measurable, predictable behavioral reaction upon natural MF fluctuations could be unambiguously proven in a mammal, and (c) high sensitivity to small changes in polarity, rather than in intensity, of MF was identified as biologically meaningful.”

I agree with the authors that their results are interesting and should be a stimulus for further research, but I do not agree that the results provide the unambiguous proof they claim. As always, replication and research focused on testing specific predictions based on the hypothesis put forward in this report, with efforts to account for alternative explanations of these observations, will be needed to determine whether the authors’ confidence in their findings is justified.

As a veterinarian, explaining science to non-scientists and interpreting the meaning of scientific research is a key part of my job. Pet owners cannot make truly informed decisions about what to do for their animal companions without reliable information they can understand. This blog arose out of my efforts to provide better information to my clients, and it has led to further efforts to inform the public, and my colleagues in veterinary medicine, about how to evaluate medical interventions and understand the scientific research we need to support making decisions for pets.

My own knowledge about how we understand health and disease has come from many years of academic study. This includes a master’s degree I will be finishing this year in epidemiology, the branch of science specifically devoted to understanding health and disease and generating safe and effective healthcare interventions. And hopefully I have developed some ability to effectively communicate about science through my academic background, my years as a veterinarian, and my work speaking and writing for the veterinary community and, of course, in this blog.

In a sense, this blog has made me part of “The Media,” as has my involvement with the American Society of Veterinary Journalists. Unfortunately, taken as a whole “The Media” does not do a very good job of covering scientific topics, and journalists seem to contribute to misconceptions at least as often as they dispel them. A newly published study looking specifically at media coverage of healthcare research illustrates this starkly.

Schwitzer GA. A Guide to Reading Health Care News Stories. JAMA Intern Med. Published online May 05, 2014. doi:10.1001/jamainternmed.2014.1359

This paper reports on a 7-year evaluation of media stories from print and electronic media of various kinds. It details a number of specific errors in how journalists often present and interpret scientific research that lead to a false understanding of what the results mean. The conclusion of the study was

After reviewing 1889 stories (approximately 43%newspaper articles, 30% wire or news services stories, 15%online pieces [including those by broadcast and magazine companies], and 12%network television stories), the reviewers graded most stories unsatisfactory on 5 of 10 review criteria: costs, benefits, harms, quality of the evidence, and comparison of the new approach with alternatives. Drugs, medical devices, and other interventions were usually portrayed positively; potential harms were minimized, and costs were ignored.

The specific kinds of mistakes made in many stories about healthcare research struck me not only because I see them all the time in the media, but because they mirror very closely exactly the sorts of mistakes made by advocates of alternative therapies. Though this study did not, unfortunately, look specifically at coverage of alternative medicine, my subjective impression is that the media makes the same sorts of errors but is even less careful and critical in coverage of this area. Pieces on veterinary medicine, in particular, are often poor quality because they are part of the “lifestyle” or “human interest” beat rather and treated as entertainment rather than being written by qualified science journalists interested in the truth about healthcare practices.

In any case, here are the major problems the study identified in media coverage of healthcare science:

Risk Reduction Stated in Relative, Not Absolute, Terms
Stories often framed benefits in the most positive light by including statistics on the relative reduction in risk but not the absolute reduction in risk. Consequently, the potential benefits of interventions were exaggerated.

While journalists are often understandably loath to talk about anything that sounds like math, it is impossible to appropriately talk about the effects of medical therapies without identifying the difference between absolute and relative risk. If you have a 1 in a million chance of developing a terrible disease, and something raises your chances to 2 in a million, that is a relative risk increase of 100%. Sounds terrible! But the thing is, at a chance of 2 in a million, you are still almost certainly not going to get that disease. And doubling your risk does not make it meaningfully more likely that you will. Such a simple distinction is critical to deciding whether medical interventions are worthwhile

Failure to Explain the Limits of Observational Studies
Often, the stories fail to differentiate association from causation.

You may have heard the saying “correlation does not mean cause and effect.” Just because two things are associated doesn’t mean one caused the other. If, for example, a study found that carrying matches in your pocket was associated with an increase in your risk of lung cancer of ten times, would that mean matches cause lung cancer? Of course not! Carrying matches may mean you’re a smoker, and smoking certainly does cause lung cancer, but the simple association between matches and cancer doesn’t mean one causes the other.

Here’s a great site that illustrates all kinds of such bogus associations. While this may not be something everyone appreciates in daily life, journalists writing about healthcare research ought to understand it.

The Tyranny of the Anecdote
Stories may include positive patient anecdotes but omit trial dropouts, adherence problems, patient dissatisfaction, or treatment alternatives.

I’ve written about anecdotes and miracle stories many times. The number one “argument” presented in the comments on this blog in defense of treatments I evaluated critically is the presentation of anecdotes that look like they show the treatment working. Anecdotes can only suggest hypotheses to test, but they can never prove these hypotheses true.

There are many reasons treatments that don’t work may seem like they do, and professionals who interpret and explain science should know anecdotes are unreliable and often misleading. While personal stories make for more interesting and emotionally appealing narratives, they should always be used carefully only to illustrate something that has been demonstrated to be true or false by more reliable evidence.

Surrogate Markers May Not Tell the Whole Story
Journalists should distinguish changes in surrogate markers of disease from clinical endpoints, including serious disease or death. Many news stories, however, focus only on surrogate markers, as do many articles in medical journals.

The bottom line for any medical treatment is whether it reduces the meaningful symptoms of disease, including the most final of all, death. It makes no difference if a therapy raises or lowers the amount of some chemical we can measure in the blood if that isn’t a clear and well-established indicator that the therapy will also reduce suffering or prevent death. Surrogate markers are, as the article suggests, overused by healthcare researchers in many cases because they are often cheaper and easier to measure than real symptoms or mortality, but they have significant limitations, and this should be made clear when talking about research using them.

Stories About Screening Tests That Do Not Explain the Tradeoffs of Benefits and Harms
Stories about screening tests often emphasize or exaggerate potential benefits while minimizing or ignoring potential harms. We found many stories that lacked balance about screening for cardiovascular disease and screening for breast, lung, ovary, and prostate cancer.

I have frequently referred to the growing appreciation in human medicine, which has not yet come very far in the veterinary field, that screening tests have risks as well as benefits, and these need to be carefully weighed. The Choosing Wisely project is a key resource for people trying to make smart decisions about screening tests, as is the web site for the U.S. Preventative Services Task Force. Both provide real evidence to help balance the risks and benefits of potential screening tests. Journalists should be aware of the limitations and pitfalls of screening and risks such as overdiagnosis and should include those considerations in stories about screening tests.

Fawning Coverage of New Technologies
Journalists often do not question the proliferation of expensive technologies.

I would add that journalists rarely question the value or evidence for alternative therapies and tend to fawn over them and their proponents more often than not. Reporting that is truly informative and useful must be thoughtful and based on assessment of the real evidence, not simply unquestioningly enthusiastic about therapies with a token quote or two from skeptics for “balance.” Drugs are not the only medical treatment to have risks, but it seems journalists are far more likely to talk about the risks of pharmaceuticals than other treatments.

Uncritical Health Business Stories
Health business stories often provide cheerleading for local researchers and businesses, not a balanced presentation of what new information means for patients. Journalists should be more skeptical of what they are told by representatives of the health care industry.

I would argue that identifying any potential bas, financial or otherwise, in a source for a news story should be an ordinary part of journalistic practice. The idea behind seeking multiple sources is not just to provide a superficial impression of balance by including opposing points of view regardless of merit but to ensure that the journalist has a comprehensive awareness of the evidence for and against the treatment they are writing about so that they can provide a useful explanation of what is known about it. The study also found, however, that journalists often don’t follow this practice.

Single-Source Stories and Journalism Through News Releases
Half of all stories reviewed relied on a single source or failed to disclose the conflicts of interest of sources. However, journalists are expected to independently vet claims. Our project identified 121 stories (8% of all applicable stories) that apparently relied solely or largely on news releases as the source of information.

There really shouldn’t be any need to point out that this is lazy and unacceptable journalistic practice and does not lead to accurate, useful information for the public.

I don’t want to suggest that there are not many excellent journalists providing accurate and informative interpretation and analysis of healthcare research. The study specifically identifies examples of stories that succeeded in avoiding the mistakes they found, and there are certainly many in the media who do a brilliant job reporting and explaining health sciences research. Hopefully, by identifying common problems and mistakes, this study will contribute to improving the quality of healthcare science journalism.