Earlier this year, I reviewed a research study claiming that dogs orient themselves to fluctuations in the earth’s magnetic field when defecating. I was asked to re-post the review on Publons, a site which publishes reviews of journal articles. The authors have posted a response there which answers some concerns I expressed in my review, but which also illustrates some of the same misconceptions about statistics and hypothesis testing that I originally discussed. I have responded both here and on the Publons forum.

The summary of the reactions of the media on our paper is very fitting and we agree. The critic of our study is, however, biased and indicates that the author did not read the paper carefully, misinterpreted it in some cases, and, in any case is so “blinded” by statistics that he forgets biology. Statistics is just a helpful mean to prove or disprove observed phenomena. The problem is that statistics can “prove” phenomena and relations which actually do not exist, but it can also “disprove” phenomena which objectively exist. So, not only approaches which ignore proper statistics might be wrong but also uncritical sticking on statistical purity and ignoring real life.

To begin with, I believe the author and I agree that statistics are easily and commonly misused in science. Unfortunately, this response seems to perpetuate some of the misconceptions about the role of statistics in testing hypotheses I discussed in my original critique.

Statistics never prove or disprove anything. Schema such as Hill’s Criteria of Causation and other mechanisms for evaluating the evidence for relationships observed in research studies illustrate the fact that establishing the reality of hypothesized phenomena in nature is a complex business that must rest on a comprehensive evaluation of many different kinds of evidence. It is unfortunate that p-values have become the sine qua non of validating explanations of natural phenomena, at least in medicine (which is the domain I am most familiar with). The work of John Ionnidis and the growing interest in Bayesian statistical methods are examples of the move in medical research to address the problem of improper use and reliance on frequentist statistical methods.

That said, these methods do have an important role in data analysis, and they contribute significantly to our ability to control for chance and other sources of error in research. The proper role of statistical hypothesis testing is to help assess the likelihood that our findings might be due to chance or confounding variables, which humans are notoriously terrible at recognizing. If we employ these tools improperly, then they cease to fulfill this function and instead they generate a false impression of truth or reliability for results that may easily be artifacts of chance or bias.

The authors accuse me of being “so ‘blinded’ by statistics that he forgets biology.” This is ironic since their paper uses statistics to “prove” something which a broader consideration of biology, evolution, and other information would suggest is improbable. Even if the statistical methods were perfectly and properly applied, they would not be “proof” of anything any more than improper use of statistics would be definitive “disproof” or the authors’ hypothesis. While I discussed some concerns about how statistics were used in the paper, my objections were broader than that, which the authors do not appear to acknowledge.

The author of this critic blames us of “data mining”. Well, first we should realize that there is nothing wrong about data mining. This is an approach normally used in current biology and a source of many interesting and important findings. We would like to point out that we have not “played” with statistics in order to find out eventually some “positive” results. And we have definitively not sorted data out. We just tested several hypotheses and always when we rejected one, we returned all the cards (i.e. data) into the game and tested, independently, anew, another hypothesis.

Though I am not a statistician, I believe there is a consensus that while exploratory analysis of data is, of course, appropriate and necessary, the post-hoc application of statistical significance tests to data after patterns in the data have already been observed is incorrect and misleading. This is what the paper appeared to suggest was done, and this would fit the definition of inappropriate data-dredging.

Note also that we performed this search for the best explanation in a single data sample of one dog only, the borzoi Diadem, for which we had most data. When we had found a clue, we tested this final hypothesis in other dogs, now without Diadem.

This was not indicated in the description of the methods provided in the original paper. If the exploratory analysis was done with one data set while the authors remained blind to the data set actually analyzed in the paper, then that would be an appropriate method of data analysis. The subsequent statistically significant results would not, of course, necessarily prove the hypothesis to be true, but they would at least reliably indicate the likelihood that they were due solely to chance effects.

This does not, however, entirely answer the concern that the study began without a defined hypothesis and examined a broad range of behaviors and magnetic variables in order to identify a pattern or relationship. As exploratory, descriptive work this is, of course, completely appropriate. But the authors then use statistical hypothesis testing to support very strong claims to have “proven” a hypothesis not even identified until after the data collection was completed. This seems a questionable way to employ frequentist statistical methods.

Let us illustrate our above arguments about statistics and “real life” on two examples. Most medical diagnoses are done through exclusion or verification of different hypotheses in subsequent steps. Does it mean that when the physician eventually finds that a patient suffers under certain illness, the diagnosis must be considered improbable because the physician has already before tested (and rejected) several other hypotheses?

This analogy is inapplicable. The process of inductive reasoning a clinician engages in to seek a diagnosis in an individual patient is not truly analogous to the process of collecting data and then evaluating it statistically to assess the likelihood that patterns seen in the data are due to chance. Making multiple statistical comparisons, particularly after one has already sought for patterns in the data, invalidates the application of statistical hypothesis testing. The fact that in other contexts, and without the use of such statistical methods, people consider possible explanations and then accept or reject them based on their observations is irrelevant.

Or imagine that we want to test the hypothesis that the healthy human can run one kilometer with an average speed of 3 m/s. We find volunteers all over the country who should organize races and measure the speed. We shall get a huge sample of data, we have an impression that our hypothesis is correct but the large scatter makes the result insignificant. So we try to find out what could be the factors influencing speed. We test the age – and find out that indeed older people are slower than younger ones, so we divide the sample into age categories, but the scatter is still too high, so we test the effect of sex, we find a slight influence, but it still cannot explain the scatter, we test the position of the sun and time of the day, but find no effect, we test the effect of wind, but the wind was weak or it was windless during races, so we find no effect. We are desperate and we visit the places where the races took place – and we find the clue: some races were done downhill (and people ran much faster), some uphill (and people ran much slower), those who ran in flat land ran on average with the speed we expected. So we can now conclude that our hypothesis was correct and moreover we found an effect of the slope on running speed. We publish a paper describing these findings and then you publish a critic arguing that our approach was just data mining and was wrong and hence our observation is worthless and that the slope has no effect on running speed at all. Absurd!

Again, this example simply describes a process for considering and evaluating multiple variables in order to explain an observed outcome, which is not the objection raised to the original paper. If the only hypothesis in a study such as described here was that at least one human being could run this fast, then a single data point would be sufficient proof and statistics would be unnecessary. However, if one is trying to explain differences in the average speed of different groups of people based on the sorts of variables mentioned, the reliability of the conclusions and the appropriateness of the statistical methods used would depend on how the data was collected and analyzed. In any case, nothing about this has any direct relevance to whether or not the data collection and analysis in the original paper was appropriate or justified the authors’ conclusions.

As I said in the original critique, this study raises an interesting possibility; that dogs may adjust their behavior to features of the magnetic field of the earth. The study was clearly a broadly targeted exploration of behavior and various features of the magnetic environment: “we monitored spontaneous alignment in dogs during diverse activities (resting, feeding and excreting) and eventually focused on excreting (defecation and urination incl. marking) as this activity appeared to be most promising with regard to obtaining large sets of data independent of time and space, and at the same time it seems to be least prone to be affected by the surroundings.” It did not apparently start with a specific, clearly defined hypothesis and prediction, so in this sense it seems an interesting exploratory project.

However, with such a broad focus, with mostly post-hoc hypothesis generation, and with a lack of clear controls for a number of possible alternative explanations, the study cannot be viewed as definitive “proof” of the validity of the explanation the authors provide for their observations, though this is what is claimed in the paper: “…for the first time that (a) magnetic sensitivity was proved in dogs, (b) a measurable, predictable behavioral reaction upon natural MF fluctuations could be unambiguously proven in a mammal, and (c) high sensitivity to small changes in polarity, rather than in intensity, of MF was identified as biologically meaningful.”

I agree with the authors that their results are interesting and should be a stimulus for further research, but I do not agree that the results provide the unambiguous proof they claim. As always, replication and research focused on testing specific predictions based on the hypothesis put forward in this report, with efforts to account for alternative explanations of these observations, will be needed to determine whether the authors’ confidence in their findings is justified.

As a veterinarian, explaining science to non-scientists and interpreting the meaning of scientific research is a key part of my job. Pet owners cannot make truly informed decisions about what to do for their animal companions without reliable information they can understand. This blog arose out of my efforts to provide better information to my clients, and it has led to further efforts to inform the public, and my colleagues in veterinary medicine, about how to evaluate medical interventions and understand the scientific research we need to support making decisions for pets.

My own knowledge about how we understand health and disease has come from many years of academic study. This includes a master’s degree I will be finishing this year in epidemiology, the branch of science specifically devoted to understanding health and disease and generating safe and effective healthcare interventions. And hopefully I have developed some ability to effectively communicate about science through my academic background, my years as a veterinarian, and my work speaking and writing for the veterinary community and, of course, in this blog.

In a sense, this blog has made me part of “The Media,” as has my involvement with the American Society of Veterinary Journalists. Unfortunately, taken as a whole “The Media” does not do a very good job of covering scientific topics, and journalists seem to contribute to misconceptions at least as often as they dispel them. A newly published study looking specifically at media coverage of healthcare research illustrates this starkly.

Schwitzer GA. A Guide to Reading Health Care News Stories. JAMA Intern Med. Published online May 05, 2014. doi:10.1001/jamainternmed.2014.1359

This paper reports on a 7-year evaluation of media stories from print and electronic media of various kinds. It details a number of specific errors in how journalists often present and interpret scientific research that lead to a false understanding of what the results mean. The conclusion of the study was

After reviewing 1889 stories (approximately 43%newspaper articles, 30% wire or news services stories, 15%online pieces [including those by broadcast and magazine companies], and 12%network television stories), the reviewers graded most stories unsatisfactory on 5 of 10 review criteria: costs, benefits, harms, quality of the evidence, and comparison of the new approach with alternatives. Drugs, medical devices, and other interventions were usually portrayed positively; potential harms were minimized, and costs were ignored.

The specific kinds of mistakes made in many stories about healthcare research struck me not only because I see them all the time in the media, but because they mirror very closely exactly the sorts of mistakes made by advocates of alternative therapies. Though this study did not, unfortunately, look specifically at coverage of alternative medicine, my subjective impression is that the media makes the same sorts of errors but is even less careful and critical in coverage of this area. Pieces on veterinary medicine, in particular, are often poor quality because they are part of the “lifestyle” or “human interest” beat rather and treated as entertainment rather than being written by qualified science journalists interested in the truth about healthcare practices.

In any case, here are the major problems the study identified in media coverage of healthcare science:

Risk Reduction Stated in Relative, Not Absolute, Terms
Stories often framed benefits in the most positive light by including statistics on the relative reduction in risk but not the absolute reduction in risk. Consequently, the potential benefits of interventions were exaggerated.

While journalists are often understandably loath to talk about anything that sounds like math, it is impossible to appropriately talk about the effects of medical therapies without identifying the difference between absolute and relative risk. If you have a 1 in a million chance of developing a terrible disease, and something raises your chances to 2 in a million, that is a relative risk increase of 100%. Sounds terrible! But the thing is, at a chance of 2 in a million, you are still almost certainly not going to get that disease. And doubling your risk does not make it meaningfully more likely that you will. Such a simple distinction is critical to deciding whether medical interventions are worthwhile

Failure to Explain the Limits of Observational Studies
Often, the stories fail to differentiate association from causation.

You may have heard the saying “correlation does not mean cause and effect.” Just because two things are associated doesn’t mean one caused the other. If, for example, a study found that carrying matches in your pocket was associated with an increase in your risk of lung cancer of ten times, would that mean matches cause lung cancer? Of course not! Carrying matches may mean you’re a smoker, and smoking certainly does cause lung cancer, but the simple association between matches and cancer doesn’t mean one causes the other.

Here’s a great site that illustrates all kinds of such bogus associations. While this may not be something everyone appreciates in daily life, journalists writing about healthcare research ought to understand it.

The Tyranny of the Anecdote
Stories may include positive patient anecdotes but omit trial dropouts, adherence problems, patient dissatisfaction, or treatment alternatives.

I’ve written about anecdotes and miracle stories many times. The number one “argument” presented in the comments on this blog in defense of treatments I evaluated critically is the presentation of anecdotes that look like they show the treatment working. Anecdotes can only suggest hypotheses to test, but they can never prove these hypotheses true.

There are many reasons treatments that don’t work may seem like they do, and professionals who interpret and explain science should know anecdotes are unreliable and often misleading. While personal stories make for more interesting and emotionally appealing narratives, they should always be used carefully only to illustrate something that has been demonstrated to be true or false by more reliable evidence.

Surrogate Markers May Not Tell the Whole Story
Journalists should distinguish changes in surrogate markers of disease from clinical endpoints, including serious disease or death. Many news stories, however, focus only on surrogate markers, as do many articles in medical journals.

The bottom line for any medical treatment is whether it reduces the meaningful symptoms of disease, including the most final of all, death. It makes no difference if a therapy raises or lowers the amount of some chemical we can measure in the blood if that isn’t a clear and well-established indicator that the therapy will also reduce suffering or prevent death. Surrogate markers are, as the article suggests, overused by healthcare researchers in many cases because they are often cheaper and easier to measure than real symptoms or mortality, but they have significant limitations, and this should be made clear when talking about research using them.

Stories About Screening Tests That Do Not Explain the Tradeoffs of Benefits and Harms
Stories about screening tests often emphasize or exaggerate potential benefits while minimizing or ignoring potential harms. We found many stories that lacked balance about screening for cardiovascular disease and screening for breast, lung, ovary, and prostate cancer.

I have frequently referred to the growing appreciation in human medicine, which has not yet come very far in the veterinary field, that screening tests have risks as well as benefits, and these need to be carefully weighed. The Choosing Wisely project is a key resource for people trying to make smart decisions about screening tests, as is the web site for the U.S. Preventative Services Task Force. Both provide real evidence to help balance the risks and benefits of potential screening tests. Journalists should be aware of the limitations and pitfalls of screening and risks such as overdiagnosis and should include those considerations in stories about screening tests.

Fawning Coverage of New Technologies
Journalists often do not question the proliferation of expensive technologies.

I would add that journalists rarely question the value or evidence for alternative therapies and tend to fawn over them and their proponents more often than not. Reporting that is truly informative and useful must be thoughtful and based on assessment of the real evidence, not simply unquestioningly enthusiastic about therapies with a token quote or two from skeptics for “balance.” Drugs are not the only medical treatment to have risks, but it seems journalists are far more likely to talk about the risks of pharmaceuticals than other treatments.

Uncritical Health Business Stories
Health business stories often provide cheerleading for local researchers and businesses, not a balanced presentation of what new information means for patients. Journalists should be more skeptical of what they are told by representatives of the health care industry.

I would argue that identifying any potential bas, financial or otherwise, in a source for a news story should be an ordinary part of journalistic practice. The idea behind seeking multiple sources is not just to provide a superficial impression of balance by including opposing points of view regardless of merit but to ensure that the journalist has a comprehensive awareness of the evidence for and against the treatment they are writing about so that they can provide a useful explanation of what is known about it. The study also found, however, that journalists often don’t follow this practice.

Single-Source Stories and Journalism Through News Releases
Half of all stories reviewed relied on a single source or failed to disclose the conflicts of interest of sources. However, journalists are expected to independently vet claims. Our project identified 121 stories (8% of all applicable stories) that apparently relied solely or largely on news releases as the source of information.

There really shouldn’t be any need to point out that this is lazy and unacceptable journalistic practice and does not lead to accurate, useful information for the public.

I don’t want to suggest that there are not many excellent journalists providing accurate and informative interpretation and analysis of healthcare research. The study specifically identifies examples of stories that succeeded in avoiding the mistakes they found, and there are certainly many in the media who do a brilliant job reporting and explaining health sciences research. Hopefully, by identifying common problems and mistakes, this study will contribute to improving the quality of healthcare science journalism.