One of the major focuses of my criticism of both science-based and alternative medicine is the failure of doctors to recognize their own limitations. There are innumerable cognitive biases and other sources of error that interfere with accurate and effective clinical decision-making. And there are many reasons why uncontrolled observations, whether by doctors or anyone else, are unreliable and inferior to controlled scientific research when trying to understand the causes of disease and the effects of healthcare treatments. The problems caused by opinion-based and faith-based medicine, including the issue of overdiagnosis and overtreatment in mainstream medicine and the persistence of ineffective treatments, especially in alternative medicine, are directly related to our tendency as individuals to trust our own judgment and beliefs far beyond their real reliability.
A new review of studies involving medical doctors illustrates this problem, and reminds us why we have to rely more on science and less on our own opinions and beliefs if we want to provide the best care for our pets and patients.
Hoffmann TC, Del Mar C. Clinicians’ Expectations of the Benefits and Harms of Treatments, Screening, and Tests: A Systematic Review. JAMA Intern Med. Published online January 09, 2017. doi:10.1001/jamainternmed.2016.8254
This review looked at studies evaluating how well MDs did at predicting the likely benefit or harm to patients of tests and treatments compared with controlled research. The findings starkly illustrated that, “clinicians rarely had accurate expectations of benefits or harms of the interventions, with inaccuracies in both directions, although they more often overestimated rather than underestimated benefits and underestimated rather than overestimated harms.”
The magnitude of this effect was impressive.
Among the studies comparing benefit expectations…most participants provided correct estimation for only 3 outcomes (11%). Of the studies comparing expectations of harm …a majority of participants correctly estimated harm for 9 outcomes (13%). Where overestimation or underestimation data were provided, most participants overestimated benefit for 7 (32%) and underestimated benefit for 2 (9%) of the 22 outcomes, and underestimated harm for 20 (34%) and overestimated harm for 3 (5%) of the 58 outcomes.
Guessing the correct benefit or risk less than 15% of the time and overestimating benefits by 32% and underestimating harm by 34% is a recipe for ineffective, even dangerous care. Though this is a study of MDs, there is no reason to think vets would do any better, and in fact it is likely that vets would perform worse than MDs and alternative medicine practitioners would do worse than those practicing science-based medicine. Vets generally have less pressure to know and conform to evidence-based standards than MDs due to less regulation and litigation. And alternative practitioners are, at best, often uninterested in scientific evidence and sometimes actively hostile to it.
The direction of the effect was also interesting, and consistent with what we know about how the human mind works. As doctors, we overestimate the benefits of our actions and underestimate the risks because we feel pressure to act and to fix things and because we need to believe we are helping our patients effectively. We are quite worried about causing harm, but we also worry more about the risks of not acting than of taking action.
The finding of more instances of clinicians underestimating harms and overestimating benefits than the opposite provides some support for the existence of therapeutic illusion (“an unjustified enthusiasm for treatment on the part of both doctors and patients,” which is a proposed contributor to the inappropriate use of interventions. Other potential contributors include the often-misleading portrayal of intervention benefits and absence of harms data in journal articles and information from commercial sources, such as pharmaceutical advertisements in medical journals.
We found much more focus on assessing expectations about harm than benefit (67% of studies measured harm expectations only) in contrast to our review5 of patient expectations where most studies (63%) focused on benefit expectations. Clinicians may be more sensitive to harming patients rather than just not providing benefit, which may stem from a fundamental concern of primum non nocere: the primary duty of doing no harm. Medicolegal concerns may also influence clinicians to place greater emphasis on the risks of not doing something rather than the risk of harm from intervening.
The response to this kind of information is not, of course, to give up on medicine. Medical care is tremendously effective at reducing suffering and death. The takeaway message is that individual observation and judgment should always be supported and informed by scientific research, which does a better job at evaluating the causes of disease and the effects of treatments than our ad hoc observations.
