One of the most common objections I see to my promotion of science-based treatments is that the scientific evidence supporting them is invalid because it is tainted by some association with Big Pharma, Big Kibble, or some other industry bogeyman. This is a convenient objection to any kind of evidence for science-based medicine or against alternative medicine because it uses elements of truth to make an ultimately invalid argument.

Financial bias is a real thing, and it does impact how we should evaluate scientific research. However, conflicts of interest are a lot more complicated than this, and our understanding and interpretation of them needs to be a lot more sophisticated than “industry funded=bulls$#t” Here is my attempt, from my VPN+ column, to take a more nuanced look at this subject.

Introduction

A key component of evidence-based medicine (EBM), and of modern life, for that matter, is critical appraisal. This means critically evaluating information before accepting or acting on it. In this time of “alternative facts” and widespread harm caused by acceptance of false or misleading information, the core EBM skill of critical appraisal is more important than ever.

In the context of evidence-based veterinary medicine (EBVM), critical appraisal refers specifically to “a formal, unbiased, systematic approach to assessing the quality and relevance of evidence presented in a paper and its applicability to decision making for our patients.”¹

As a clinician, before I decide whether to believe the results or conclusions of a published research study, or utilize them in caring for my patients, I have a responsibility to decide for myself how reliable the results are, how likely they are to represent the truth, and whether they are applicable to my patients.

All too often, vets skim the abstract or discussion section of a paper, find the main conclusion, and then take that on faith and use it to guide their practice. Unfortunately, much of what is published in the scientific literature turns out to be more complicated than it first appears, irrelevant to many patients in the world of primary care, or simply wrong. Critical appraisal is the last line of defense between our patients and unsafe or ineffective treatment based on bad information.

There are many aspects of research publications that must be evaluated in the course of a thorough critical reading. The focus of this column is conflict of interest (COI). This is an issue frequently mentioned in guides to critical appraisal, and often used to dismiss the results of published research, but it is an element of critical appraisal that is especially problematic and for which clear guidelines are not available.

What is a conflict of interest?

Even the definition of a COI is challenging and subject to debate.^2–4 I find it useful to define a COI broadly as any personal, institutional, or financial relationship that might influence the beliefs, judgements, or actions of an investigator during the design, conduct, analysis, and reporting of scientific research. This helps to distinguish a COI from mere personal belief, but it goes beyond the common and excessively narrow view of COIs as exclusively about financial interests.

A COI is simply one category of bias that can influence the outcome of scientific research. In ordinary use, the term “bias” has negative connotations as it mostly refers to various forms of prejudice or discrimination against groups of people. However, in the parlance of EBM, “bias” simply means any factor that systematically distorts the results of scientific research away from the truth.

If I flip a coin that is unevenly weighted 100 times and get heads 80 percent of the time that is a type of bias. If a researcher chooses only male dogs to study the effects of a drug, that is a type of bias that can lead to a result consistently different from the true effects of the drug on dogs in general. If a researcher has a particular belief that affects how they choose their hypothesis, design their study, analyze the data, and decide whether or not to publish, that is a type of bias, as well.

It should be clear from this definition and examples that bias is ubiquitous in all research. It is not a sign of bad intentions or incompetence; it is an unavoidable consequence of the operation of the human brain and the imperfection of scientific methods. None of us is without bias if we have any beliefs or distinctive individuality at all. The methods of science exist to help compensate for some of the most common and troublesome sources of bias that arise from the operation of the human brain. This is a pretty uncontroversial view in the fields of EBM and the philosophy of science.

I will also make the potentially more controversial claim, however, that none of us is truly free of any conflict of interest either. Science is a community process, rarely conducted in isolation from other people or institutions, and our relationships inevitably affect our beliefs and practices as scientists. Despite the implications of the word “conflict” in conflict of interest, I believe it is useful to think of a COI as simply a type of bias, and as such, morally neutral in most cases. Just as the term bias lacks pejorative connotations in EBM, the term COI should not be seen as implying malfeasance or bad faith.

Of course, deliberate actions to alter or even fabricate scientific data motivated by financial interests, career considerations, or other types of COI relationships are clearly unethical. However, most COI, like other types of bias that trouble scientific research, involves unconscious and unintentional skewing of research results in a direction influenced by our relationships and beliefs. A researcher can be perfectly honest and genuine in attempting to produce objective, reliable data and still have a COI that influences the results of their research.

What constitutes a COI?

The majority of research into COI and how it influences medical research outcomes focuses on financial COI.^2,4 If a researcher stands to benefit financially in some way from a certain study outcome, they are considered to have a COI. This may involve financial gain from the success of a product or service being tested, or the effect of research outcomes on the fortunes of a commercial entity, or other institution the researcher is affiliated with.

The classic example of financial COI is research funded by pharmaceutical companies. It has been pretty consistently demonstrated in human biomedical research industry funding of clinical studies is associated with more favorable study outcomes than is funding by government or nonprofit organizations.^3,4 There is limited evidence in veterinary medicine, but it seems likely a similar industry funding bias exists in veterinary clinical trials.⁵

This relationship is often used to dismiss industry-funded research as hopelessly biased, and to suggest there is deliberate manipulation of all such research for financial gain. Such dismissals, not surprisingly, often come from individuals critical of conventional medical treatments and interested in promoting alternatives. However, it is no more accurate or useful to suggest funding bias is simply fraud or it invalidates all research connected with pharmaceutical or other commercial entities than it is to deny such bias exists at all.

Funding bias is a serious problem, especially given the paucity of noncommercial funding sources for veterinary research. However, it is very rarely the case that it manifests as deliberate fabrication or manipulation of data on the part of iindividuals hoping to improve their financial fortunes. Such deliberate cheating would, perversely, be easier to detect and prevent than the real problem, which is unconscious bias embedded in the perspective of researchers, which influences the questions asked, the design and conduct of studies, and the analysis and reporting of results.

Researchers are typically affiliated with institutions; mostly universities or commercial companies. People tend to gravitate toward institutions populated by like-minded individuals with similar views, and we tend to incorporate the views of our close colleagues or mentors into our own perspectives on scientific questions. Similarly, commercial organizations tend to hire researchers and fund research that aligns with their commercial goals.

For example, a scientist interested in the possibility the anti-inflammatory effects of non-steroidal anti-inflammatory drugs (NSAIDs) might suppress the development of cancer is likely to seek work or funding from companies developing such drugs, not from an herbal medicine company that promotes natural remedies as safer than NSAIDs.

Additionally, a nutritionist concerned about the negative health effects of extruded commercial diets isn’t likely to seek or find support for research seeking to identify such harms from a company selling such diets, and they will likely end up working with like-minded researchers and seeking funding from sources with aligned interests.

Finally, a doctoral student in need of mentoring, funding, and help starting a career is very likely to share many of the views of their academic advisors, at the end of their training at least, even if not at the beginning.

These are all examples of the natural alignment of views that tends to occur when people join together in institutions to participate in scientific research. Funding bias, like other forms of COI, is more likely to be an expression of the shared perspectives of individuals grouped in institutions than simple fraud. This does not, of course, mean it is not a serious problem that skews research results away from the truth. The fact commercial organizations choose what research to conduct or support, and influence the way that research is designed, conducted, and reported, can lead to a body of evidence significantly mistaken or misleading even without any fraudulent intentions or actions.

However, it is worth noting the same type of alignment of views, leading to systematically biased research results, can occur when the alignment is fostered by ideological, political, cultural, and other nonfinancial factors. Research supported by pharmaceutical companies is more likely to be favorable to their products than independently funded studies, but research on homeopathy conducted by homeopaths and published in alternative medicine journals is also much more likely than not to be positive.^6,7 Studies of acupuncture conducted in China, where the practice is widely embraced by the populace and the government, are much less likely to report negative results than acupuncture studies from countries where the practice is not part of the shared cultural and political history.^8–10

While financial incentives and commercial relationships form an important type of COI, we must recognize the problem of COI is seldom simplistically reducible to deliberate skewing of results in the interest of making money. The ways in which we identify and mitigate COI and incorporate it into our critical appraisal will be different, and more effective, if we take the broader view of COI as involving the influence of personal and institutional relationships on belief and judgement rather than simply following the money.

How does a COI bias research?

When viewed in this larger frame, conflicts of interest can be identified as the potential influences of institutional and personal relationships on research results at every stage of the scientific process. Which questions we choose to ask as scientists and how we go about answering them are related to our context and relationships.

As a general practitioner, for example, I have investigated factors influencing the risk of complications from routine neutering surgeries at my practice because this was a relevant subject for me and my colleagues in a way it likely would not be for an academic surgeon. My study was conducted as a clinical audit, not a randomized controlled trial, because it was part of an in-house quality improvement process at my hospital, not an academic research project. The data collection tools (simple questionnaires) reflected the practical realities and interests of the doctors participating in the study. The examination of how various types of suture material might influence complication rates included only those types of suture we use at our practice, and the neutering techniques compared were those our doctors used, even though there are other materials and techniques we could have considered. These and 100 other choices made during the research project reflected bias associated with my personal and institutional relationships.

One might be tempted to argue this isn’t truly an example of a COI. Given the emphasis on financial incentives in discussions of conflict of interest, one could claim there is not a COI for me to disclose if I publish the results of my study because I don’t have an obvious financial stake in the outcome.

Of course, one could imagine a possible financial or personal incentive. Perhaps clients would stop choosing to have their pets neutered at my practice if we reported a high rate of complications? Perhaps my boss would penalize doctors with higher complication rates, and I might be tempted to adjust my findings depending on my relationship with my colleagues?

However, my point is such narrow personal motives are only a limited subset of the ways in which our personal and professional affiliations influence our scientific research efforts, and excessive focus on these types of influences leads us to overlook other important and relevant factors.

I suspect the types of COI-related bias found in research conducted at commercial and academic institutions resembles this example more than the cliché of shadowy figures in industry deliberately manipulating or hiding data to make their companies’ products look good. I have no doubt that happens, of course, but it is far less common than a subtle, creeping bias introduced by a thousand small choices made in an environment where most people have the same perspective and beliefs.

Regular readers will know, of course, in the last year I have begun working in a biotechnology company developing drugs to extend lifespan and healthspan in dogs. Is this a conflict of interest in terms of writing about COI? Sure! Of course, I work for a company with an exemplary ethical and scientific culture, and I wouldn’t have taken the job if I didn’t believe this to be true. And part of my job there is to be the nagging voice of our collective conscience and push for the best possible science within the constraints of regulations, funding, and the ultimate goal of developing successful clinical therapies. However, my perspective is absolutely influenced by the context in which I work and the ethos of my work community.

I have seen how the regulatory and economic framework of a biotech startup differ from the clinical and academic research contexts. The questions we ask, how we frame them, the studies we design to answer them, and how we handle data collection and analysis are all influenced by this context, just as my research in private practice and the research of any university veterinarian is influenced by those environments and relationships. That is why I rush to disclose this relationship any time I talk or write about any scientific subject.

The interesting and important question, then, is not whether a COI exists for a given research project, since it almost inevitably does, but what do we do about it?

How should COI be considered in critical appraisal?

In the past, a potential COI was largely ignored, with the lofty assumption that because scientists had integrity, their work would not be influenced by such relationships. As I’ve already argued, I believe we are all influenced by our relationships in subtle ways against which good intentions are not an effective defense. This has become difficult enough to deny that it is now more common for a COI to be handled through disclosure. Journals, funding bodies, regulators, and others with some authority over scientific work typically require scientists to disclose any potential COI relationships (e.g. JAVMA,JVIM, JVECC, JAAHA). Such disclosure is voluntary and typically limited to financial COIs, which is problematic, but at least it is easier now to know when such relationships exist.

Sometimes journals take failure to disclose potential COI seriously. The Journal of Veterinary Internal Medicine recently retracted a paperbecause the authors failed to disclose an obvious financial COI. On the other hand, I wrote last Octoberabout an undisclosed COI that could have potentially significant relevance for interpretation of a narrative review, and in that case the journal chose to largely ignore the omission when it was pointed out by readers.

The difficult question for the critical consumer of science, however, is what to do with information about COIs when it is disclosed. The sad reality is most often disclosed COIs are ignored entirely (if we agree with the conclusions of the research or share the general perspective of the authors, or simply because we don’t believe “good scientists” can be biased¹¹), or used as a pretext to completely discount the results (if we disagree with the conclusions or have a conflicting perspective on the issues).

To again pick on the extreme case of homeopathy, homeopaths accept the shoddiest and most obviously biased studies as probative when they show efficacy for homeopathy, and they reject any research conducted or conducted by science-based investigators or funded by any kind of entity other than a homeopathy manufacturer.⁶ (Not surprisingly, of course, I have been accused of exhibiting the same type of bias in reverse, which is fair, though I think the record of my public critical appraisal of research studies of all kinds over the years doesn’t support this claim). Sadly, veterinary students who should know better react in a similar way, discounting their potential susceptibility to bias from associations with the pharmaceutical industry.¹¹

Veterinarians who want to read research studies critically are justly confused about how to incorporate CI into their assessment, and there are few resources available to help them^12,13. My perspective is a lot of the problem stems from our view of research evidence as binary. We believe a study either proves or disproves the hypothesis it is testing; a drug either works or doesn’t work; a proposed etiology either does or does not cause a disorder. We think the results must be black or white, and we further believe we can make such judgements for every single study independently. These beliefs are mistaken and inconsistent with how science really functions.

A more useful approach to integrating COI into critical appraisal, and into our overall assessment of what is true and false in medicine, is a philosophically Bayesian manner. Though the mathematic details of Bayesian analysis are complex, the underlying principles are straightforward^14,15. They suggest that we should assess the reliability of evidence and conclusions in a research study as follows:

1. Establish a likelihood of the hypothesis or claim being true or false based on what we already known.
2. Evaluate the data presented and assess all the usual strengths and weaknesses.
3. Shift our estimate of the likelihood the hypothesis or claim is true to a degree proportional to the strength of the evidence in the paper.

This is really much simpler than even this stripped down description. It just means COI is only one factor in our overall weighting of the evidence in a given study, and each study is only one bit of evidence in our overall assessment of every claim or hypothesis. There is a continuum of confidence in both the study results and their meaning for our beliefs about whatever subject they address, not a binary state of true or false, pure or biased.

When I read a paper on, for example, a new drug, and I see it was funded by the company making the drug, this reduces my confidence in the conclusions slightly (especially if the conclusions are favorable to the company’s interests). However, if the study was conducted by independent researchers insulated from direct influence by company employees, and if the bias control methods are rigorous, that COI has a pretty small impact on my confidence in the study. In contrast, if the study was conducted in a company facility by company employees using subjective outcome measures and with poor methodological control for bias, the COI will significantly undermine my confidence in any positive results.

The details of how we evaluate the significance of potential COI will vary with the specific research project. The important takeaway here is we shouldn’t use COI as a sole reason to accept or reject evidence. We should consider potential COI in light of both the design and conduct of the specific study, and the potential for researcher bias based on the larger context.

A good example is a recent study of the influence of diet during puppyhood on the risk of canine atopic dermatitis (CAD) developing later in life, which appeared in JVIM. I have written a detailed critical appraisal of this study in my regular Veterinary Practice News column, and contributed to a letter to the editor of JVIM detailing various concerns about the paper.

The paper ultimately concludes, with some caveats, feeding raw diets to puppies may protect them against developing CAD as adults. The authors declared no COI. As required by JVIM, they did declare sources of funding. These included both manufacturers of raw diets and Joseph Mercola, an infamous proponent of raw diets and alternative medicine and an anti-vaccine activist repeatedly warned by the Food and Drug Administration (FDA) for promoting COVID-19 misinformation and selling unproven treatments.^16,17

The research group, furthermore, has an established history of publishing research promoting raw diets and claiming detrimental health effects from conventional commercial diets. The senior author, an academic researcher with some minor ties to commercial pet food companies, has specifically identified financial COI as a problem in raw diet research, saying, “[raw food is] not really researched in universities. Most universities get sponsored by these big billion-dollar companies, and you don’t really want to step on their toes, I guess. But, I think that’s not really ethical.”¹⁸

This is a great example of what I would consider a COI that does not involve direct financial interests or any intentional malfeasance. The researchers have personal and professional affiliations with individuals and institutions committed to the hypothesis raw diets are beneficial and conventional pet foods are harmful. They are undoubtedly experienced and ethical scientists, and they are unlikely to reap any significant financial gains from promoting this idea. Yet, in the context of their history, and the specific work in this study, their affiliations still amount to a COI that must be considered in weighing the reliability of the evidence they provide.

The bottom line

The more interesting I find a subject, the more I write. Not a great habit in today’s TL;DR culture! (That’s “too long, didn’t read.”)

1. Conflicts of interest are not just about money! All personal, financial, and institutional relationships influence our perspective in ways that can bias the research we do.
2. Everyone has some type of conflict of interest. Being smart and ethical doesn’t prevent this from influencing the research we do.
3. Every possible conflict of interest should be reported when we publish scientific research.
4. Conflict of interest alone does not justify uncritical rejection of scientific evidence.
5. We should weigh the significance of possible conflicts of interest in the context of the potential for bias to influence study results and the efficacy of bias control methods employed in the study.
6. No single study ever proves or disproves anything (OK, maybe once in a while, but not often!).
7. We should assess individual studies and the overall evidence for or against specific ideas in a Bayesian manner, on a continuum of confidence, not with a binary true or false scheme.

References

1. Pinchbeck GL, Archer DC. How to critically appraise a paper. Equine Vet Educ. 2020;32(2):104-109.
2. Johnson C. Conflict of Interest in Scientific Publications: A Historical Review and Update. J Manipulative Physiol Ther. 2010;33(2):81-86.
3. Mandrioli D, Kearns CE, Bero LA. Relationship between research outcomes and risk of bias, study sponsorship, and author financial conflicts of interest in reviews of the effects of artificially sweetened beverages on weight outcomes: A systematic review of reviews. PLoS One. 2016;11(9).
4. Institute of Medicine (US) Committee on Conflict of, Interest in Medical Research, Educationand P. Committee on Conflict of Interest in Medical Research, Education, and Practice. Washington, D.C.: National Academies Press (US); 2009. https://www.ncbi.nlm.nih.gov/books/NBK22926/. Accessed January 9, 2022.
5. Wareham KJ, Hyde RM, Grindlay D, Brennan ML, Dean RS. Sponsorship bias and quality of randomised controlled trials in veterinary medicine. BMC Vet Res. 2017;13(1):234. 9
6. Cukaci C, Freissmuth M, Mann C, Marti J, Sperl V. Against all odds—the persistent popularity of homeopathy. Wien Klin Wochenschr. 2020;132(9-10):232-242.
7. Ernst E. A systematic review of systematic reviews of homeopathy. Br J Clin Pharmacol. 2002;54(6):577-582. http://www.ncbi.nlm.nih.gov/pubmed/12492603. Accessed November 12, 2018.
8. Vickers A, Goyal N, Harland R, Rees R. Do certain countries produce only positive results? A systematic review of controlled trials. Control Clin Trials. 1998;19(2):159-166.
9. Ma B, Qi G, Lin X, et al. Epidemiology, Quality, and Reporting Characteristics of Systematic Reviews of Acupuncture Interventions Published in Chinese Journals. J Altern Complement Med. 2012;18(9):813-817.
10. Wang Y, Wang L, Chai Q, Liu J. Positive results in randomized controlled trials on acupuncture published in chinese journals: a systematic literature review. J Altern Complement Med. 2014;20(5):A129. www.liebertpub.com. Accessed November 14, 2018.
11. Dowers KL, Schoenfeld-Tacher RM, Hellyer PW, Kogan LR. Corporate Influence and Conflicts of Interest: Assessment of Veterinary Medical Curricular Changes and Student Perceptions. J Vet Med Educ. 2015;42(1):1-10.
12. Lundh A, Boutron I, Stewart L, Hróbjartsson A. What to do with a clinical trial with conflicts of interest. BMJ evidence-based Med. 2020;25(5):157-158.
13. Resnik DB, Elliott KC. Taking financial relationships into account when assessing research. Account Res. 2013;20(3):184-205.
14. Homwong N, Hunprasit V, Marthaler D, et al. A Bayesian approach for inductive reasoning to clinical veterinary medicine: The math of experience. J Vet Med Anim Heal. 2015;7(10):308-316.
15. Gardner IA. The utility of Bayes’ theorem and Bayesian inference in veterinary clinical practice and research. Aust Vet J. 2002;80(12):758-761.
16. Satija N, Sun L. A major funder of the anti-vaccine movement has made millions selling natural health products. Washington Post. December 20, 2019.
17. Alliance for Science. FDA warns Mercola: Stop selling fake COVID remedies and cures.
18. Habib R, Becker K. The Dog Cancer Series: Rethinking the Cancer Epidemic Vol. 1, Chapter 4 (Transcript). 2018:73.