While rigorous scientific research is the best way to evaluate most proposed medical treatments, there are circumstances in which scientific study can actually be quite misleading. Studies of implausible therapies using unreliable methods rife with uncontrolled bias serve only to create the false impression of scientific legitimacy for useless practices. The bulk of the research literature concerning homeopathy illustrates this phenomenon quite well.
One of the clearest characterizations of this problem is Dr. Harriet Hall’s description of the phenomenon of Tooth Fairy Science. Extensive controlled research can be done to evaluate the behavior of the Tooth Fairy. The relative monetary value of different teeth from children of different ages or genders, the geographic distribution of Tooth Fairy activity, and many other related variables can be analyzed, complete with complex statistics. At the end of the day, however, all this data proves nothing because the underling phenomenon being studied is imaginary. The same is often true of research into alternative medical therapies
The latest example of such Tooth fairy science is a study purporting to show that chiropractic manipulation prevents spinal spondylosis deformans in dogs.
Halle KS, Granhus A. Veterinary Chiropractic Treatment as a Measure to Prevent the Occurrence of Spondylosis in Boxers. Vet Sci. 2021 Sep 17;8(9):199. doi: 10.3390/vetsci8090199. PMID: 34564593; PMCID: PMC8473340.
Spondylosis deformans is a condition in which new bone forms gradually in the spinal column of dogs over years. It is very common in dogs, and does not appear to cause pain or other clinical symptoms, though it can reduce the flexibility of the spine in advanced cases. Spondylosis is believed to occur due to small tears in the cartilage disks between vertebrae that, which cause inflammation and perhaps excess movement, leading to reactive bone formation. There are also genetic influences, and the condition is moderately heritable in some breeds, such as Boxers.
The theory presented in this paper is that spondylosis is caused by decreased flexibility in the spine, and that chiropractic assessment and treatment can detect this excess rigidity and correct it. The study was conducted to prove this by comparing pairs of littermates evaluated and treated by a veterinary chiropractor monthly from 8 weeks to 1 year of age. The authors reported that spondylosis was less common in the treated dogs.
There are many, many issues with this study that make its conclusions unreliable and not useful for evaluating the effects of chiropractic treatment in dogs. The most obvious, which I’ve already mentioned, is that the study is testing an imaginary method of diagnosis and treatment. Chiropractors have never been able to demonstrate that the abnormalities they claim to treat exist or that they are able to reliably detect them by physical examination.1–3 Finding supposed differences in the flexibility of specific spots in the spine is an entirely subjective process, and historically chiropractors haven’t been able to convincingly demonstrate that what they claim to identify is real.
Similarly, the physical manipulation employed by chiropractors, thee low-amplitude, high-velocity thrust, hasn’t been shown to have meaningful beneficial effects despite decades of trying to prove this. The best we can say about chiropractic in humans is that it might have some small subjective benefits for back pain, no better than many other therapies, and even this rests on weak evidence.4 There is even less reason to believe there are meaningful effects in veterinary species. The idea that a short, hard shove against the bones of the spine once a month based on an unreliable subjective assessment of how flexible they are is going to have meaningful effects on health is pretty far-fetched, and it would take an extraordinary body of evidence to validate such a claim.
This paper clearly does not constitute such a body of evidence. Despite plenty of opportunities for residual bias in the study design the ultimate findings only barely reached statistical significance for one dubious measure of effect. The authors report that occurrence of spondylosis was lower in treated dogs with a p-value of 0.0478, with 0.05 being the cutoff for statistical significance. None of the other statistical analyses showed any significant results.
While a thorough explication of the methodological issues with the study is unlikely to be of interest to anyone without an unusual love of epidemiology, I will list the most significant so that the limitations of this piece of evidence can be seen. Ultimately, the fact that an imaginary diagnosis and treatment narrowly showed an effect in a study specifically designed to do so and run by investigators with a deep ideological commitment to the value of chiropractic is a classic example of how Tooth Fairy Science serves primarily to create the illusion of scientific legitimacy for therapies that are instead fundamentally pseudoscientific.
- Placebo Control, Blinding, Randomization
One way of trying to minimize the impact of variables other than the treatment being tested on the outcome is to have patients randomly assigned to get either the treatment or an indistinguishable placebo, and to make sure that no one knows which patient is getting which so that our beliefs and expectations are less likely to influence how we evaluate or treat patients in each group. This control group in this study were littermates of the treated dogs who received no chiropractic treatment. There was no placebo control, and clearly the investigator knew which puppies were treated and which weren’t. It is likely that the owners also knew, and while the people reading the x-rays to look for spondylosis were apparently blinded to the treatment, it doesn’t appear the individuals doing the statistical analysis were (it isn’t made clear who this was or what potential biases they might have had). And while the study is described as “randomized,” there is no discussion of how this was done. Overall, then, it isn’t clear that most of the core, routine methods for controlling bias and error in a scientific study were used, and one of the most important, blinding to treatment, was not.
- The subjects were recruited by convenience over an 11-year period. This too opens the door to lots of small decisions which, in aggregate, could influence which dogs are chosen and which group they are placed in, all of which can add up over such a long time to influence the ultimate outcome of the study. This is not at all standard scientific methodology.
- Spondylosis was not found to be associated with sex or the presence or absence of the condition in parents. This is in conflict with most previous studies of this condition, which find both an influence of sex and a relatively high heritability. This suggests this study either looked at an unusual population or the methods led to atypical findings.
- The dogs were assessed for spondylosis at 1 year of age. However, this condition is progressive over time and frequently emerges later in life, so whether the findings at one year were representative of the actual emergence of the condition in these dogs over the majority of their lifetime is unclear.
- The likelihood of spondylosis varies at different sites in the spinal column. Rather than picking a particular site to treat and evaluate, the owners lumped all spondylosis at any site into “Yes” or “No” categories. This can easily misrepresent the extent and severity of the condition. In fact, the authors note that not doing this led to a statistically non-significant result. Given that the statistical analysis was not apparently done blinded, this is a perfect example of how manipulation and statistical testing of data after a study can create a significant result even if the data don’t clearly support a real difference between the groups.
- The study was described as a “randomized case-control” study. There is actually no such thing, and this is more properly described as a randomized clinical trial with a variation of block randomization by litter. The terminology itself is not relevant to the outcome or conclusions, of course, but it does suggest that the authors may not have extensive familiarity with designing and conducting clinical research studies.
Most importantly, spondylosis is not a cause of significant clinical problems in a majority of dogs, and nearly all dogs will have it to some extent as they age. The very best possible outcome of this study would have been a decrease in the occurrence of something that isn’t a real problem. Chiropractors, of course, would still consider that a success since it would support their claims that what they are doing has real effects on patients. However, this study does not provide reliable evidence that this is true. It certainly does not warrant ignoring the decades of research that have failed to find that chiropractic is a real treatment for any real physical problem.
References
1. Mirtz TA, Morgan L, Wyatt LH, Greene L. An epidemiological examination of the subluxation construct using Hill’s criteria of causation. Chiropr Osteopat. 2009;17(1):13. doi:10.1186/1746-1340-17-13
2. Hestbaek L, Leboeuf-Yde C. Are chiropractic tests for the lumbo-pelvic spine reliable and valid? A systematic critical literature review. J Manipulative Physiol Ther. 2000;23(4):258-275. http://www.ncbi.nlm.nih.gov/pubmed/10820299. Accessed November 18, 2018.
3. French SD, Green S, Forbes A. Reliability of chiropractic methods commonly used to detect manipulable lesions in patients with chronic low-back pain. J Manipulative Physiol Ther. 2000;23(4):231-238. http://www.ncbi.nlm.nih.gov/pubmed/10820295. Accessed November 18, 2018.
4. McKenzie B. Placebos for Pets: The Truth About Alternative Medicine for Animals. Ockham Publishing; 2019.
