Jon Jureidini, Leemon B McHenry, The Illusion of Evidence-Based Medicine: Exposing the crisis of credibility in clinical research. Adelaide, Australia: Wakefield Press, 2020, 318p. ISBN: 978-1-74305-724-7, Price: $28.66.

Evidence-based medicine (EBM) emerged from the purest intentions. Randomised controlled trials (RCTs) should examine what works best and which adverse effects emerge during treatment. RCT results should guide treatment, leading to widespread adoption of the best, safest treatments. EBM is now widely accepted in medicine, including by the drug industry, which seamlessly incorporated EBM into sales pitches. Indeed, it is hard to imagine a modern drug marketing campaign that does not feature RCT results.

The drug industry often touts its commitment to research for the sake of benefiting patients. Internal drug industry documents suggest a different motivation for research. An internal document from Pfizer states that research is done to “Optimize our ability to sell Zoloft [sertraline] more effectively… the purpose of data is to support, directly or indirectly, marketing of our product (1).” An Eli Lilly document states that the company should: “Develop scientific research and publications plan that enhances credibility of the new positioning and enables the achievement of the ideal positioning…Mine existing data to generate and publish findings that support the reasons to believe the brand promise (1).” These are just two of many such examples. In their excellent book, The Illusion of Evidence-Based Medicine, Jon Jureidini and Leemon McHenry examine what happens when RCT data fail to support the brand promise (2).

The authors are uniquely well-qualified to explore the shortcomings of EBM. Jon Jureidini, Professor of Psychiatry and Paediatrics at the University of Adelaide and Leemon McHenry, Emeritus Lecturer of Philosophy at California State University, Northridge, have viewed thousands of pages of internal pharmaceutical company documents that emerged from litigation against GlaxoSmithKline (GSK) and Forest Laboratories regarding their antidepressants. I have also independently viewed many of the same documents as well as documents from other firms (including Eli Lilly, Janssen, AstraZeneca, Pfizer, and Parke-Davis). Readers should know that Jureidini and McHenry have been involved in legal cases against drug manufacturers and have received compensation for their work. I have no involvement with legal actions against pharmaceutical firms and find the authors’ interpretations to be accurate and reasonable.

Misreported antidepressant studies

Based largely on internal documents, the authors describe problems with two industry-funded antidepressant RCTs. Study 329 examined paroxetine and was funded by SmithKlineBeecham (now GSK); and CIT-MD-18 was a citalopram trial funded by Forest Laboratories. Internal communications show that both sponsors knew the efficacy results were negative, yet subsequent journal articles and marketing efforts claimed treatment efficacy. Adverse events were also mischaracterised and underreported. Marketing clearly trumped science, which is stated bluntly in several internal communications described by Jureidini and McHenry.

GSK’s website trumpets its “long-standing” pledge on data transparency, stating that they release patient-level individual data from trials and publish all of their clinical study reports (trial summaries that have more information than journal articles) online (3). Jureidini and McHenry are not impressed. They note that GSK’s sharing of data was not driven by a commitment to open science. Rather, it was mandated by a legal settlement with the state of New York regarding GSK covering up unfavourable efficacy and safety results in paroxetine studies (p 54). In 2015, Study 329 became one of very few industry-sponsored trials to have its raw data re-analysed by independent researchers who were given access by its sponsor (4). Jureidini was a member of this team. The efforts of the Study 329 re-analysis team are truly remarkable. GSK – the self-proclaimed champions of data access – first delayed releasing data to the independent research team. Data were finally made available through a “data periscope” that allowed remote access to raw data forms (with identifying information removed) without allowing saving or printing of the forms; this made the work of the independent researchers much more time-consuming. The re-analysis found that paroxetine completely lacked efficacy (4) whereas the original, corporate-sponsored Study 329 manuscript had (falsely) concluded that the drug was effective (5). Further, the original paper substantially underreported suicidal thoughts and behaviours and sometimes referred to these using the euphemism of “emotional lability” rather than more accurate, specific, and troubling terms describing these adverse events.

The Illusion of EBM describes several internal documents on GSK’s and Forest’s efforts to rehabilitate negative data for marketing purposes. For instance, several participants in CIT-MD-18 were provided with unblinded study medication. According to the study protocol, their data were unusable. Yet, the primary outcome only generated statistically significant results if the ineligible participants were included – so they were left in the analysis, rendering the results “positive”. The journal article which reported the study results did not mention this problem (6). Forest’s communications with the Food and Drug Administration underplayed the violation of blinding, which Forest said, in an understatement, only had “potential to cause bias.” Administering unblinded medication does not merely have the potential to cause bias – it clearly invalidates the participants’ data. The “potential to cause bias” statement was referred to as a “masterful euphemism” in an internal email. The euphemiser then stated that “part of my job is to create masterful euphemisms to protect Medical and Marketing.” Jureidini and McHenry include several such disturbing anecdotes from internal documents.

The detailed, disturbing descriptions of Study 329 and CIT-MD-18, in Chapter 2 and throughout the book, are its highlights. The authors also convincingly describe problems across the entire academic-pharmaceutical industry-“scholarly” publishing complex which have transformed evidence-based medicine into marketing-based medicine. The topics are not particularly original – this ground has been well-covered by authors such as David Healy, Carl Elliott, John Abramson, and Marcia Angell. Indeed, many respected scholars who originally supported EBM have since sounded alarm bells. For instance, the prolific and well-respected researcher John Ioannidis was once a strong proponent of EBM. His enthusiasm has dampened. He has stated that “…corporations should not be asked to practically perform the assessments of their own products. If they are forced to do this, I cannot blame them, if they buy the best advertisement (i.e., ‘‘evidence’’) for whatever they sell (7).” Jureidini and McHenry quite clearly assemble the many puzzle pieces which demonstrate that EBM has been hijacked by drug marketers. The Illusion of EBM provides strong evidence that the drug/device industries exert undue influence over the platforms which disseminate scientific data, including scientific journals and conferences, relevant websites, and the medical education industry. In each of these forums, drug firms oversell efficacy and overlook safety or tolerability concerns.

Ghosts

Jureidini and McHenry’s description of ghostwriting and the ghost management of the entire research process are on target and well worth reading, even among those who are familiar with these topics. Early in a product’s life cycle, drug firms create publication plans to disseminate research findings in the most influential manner. What data should be published in which journals to create the strongest perception that a product is supported by Evidence? Medical communications companies are well-paid by the drug industry to ensure that data are quickly rendered into effective manuscripts which are cited as evidence of drug efficacy and safety (8). Medical writers ensure that the “brand promise” is fulfilled by their manuscripts – demonstrating that sponsored drugs are fulfilling the promises of EBM.

The medical writing and journal publishing industries have a unique view of ghostwriting. If the name of a medical writer who wrote the first draft of the paper is acknowledged in a footnote as having provided “editorial support”, then no ghostwriting occurred. Yet dictionary definitions and common sense indicate that a paper mainly written by a person who is not listed as an author is ghostwritten. Listed study authors are mostly key opinion leaders (KOLs), academics whose beliefs align with industry and who typically receive substantial consulting fees, speaking fees, and/or payment in exchange for research work. KOLs are selected by industry because their views align with those of the sponsor, and because they are “opinion leaders” due to their publication record or prestigious academic affiliation. These KOLs are quite busy people who may not have time to substantially contribute to the many manuscripts that bear their name. Ghostwriters are handy at keeping the manuscript on schedule.

On its face, medical ghostwriting is outrageous for crediting “authors” with papers they didn’t write, based upon studies they did not design, and interpretations of data they did not make. Recruiting participants, providing some administrative oversight, and making minor edits to a paper may not justify authorship. But as Jureidini and McHenry point out, the bigger problems are that “authors” typically lack access to raw data and rely upon industry’s own data analyses and interpretations (Chapter 4). Largely, the sponsor controls the data, how they are analysed, and which data appear in a manuscript. Alastair Matheson, a former medical writer, has argued that academic “authors” make such unimportant contributions on many manuscripts as to make them disposable or interchangeable (9). A manuscript’s content varies somewhat depending on the particular KOLs involved with the paper. But the final message, that the product is supported by data remains fully intact regardless of the “authors,” who have been well-vetted by the sponsor beforehand. Matheson argues that “If a project is instigated and funded by a company and its data are privately owned, then it is a commercial project, and by means both of authorship and other attributive devices, it should be presented clearly to readers as commercial, not the ambiguous, supposedly academic-led fare that is a staple of medicine’s intellectual diet (9)”. In other words, these supposed bastions of science – published RCTs in prestigious medical journals – are more of a commercial product than a scientific effort. The imprimatur of academic opinion leaders adds little to the scientific strengths or weakness of industry-funded RCTs. The “authorship” of KOLs is mainly designed to wrap the corporate product (the published manuscript) in an academic package to reduce the readers’ perception of corporate influence.

Journals

A naïve observer might look for scientific journals to rein in the drug industry. But journal publishers are deeply dependent on the drug industry for revenue. Drug firms order large quantities of journal article reprints for marketing purposes (10); further, a slew of drug advertising in medical journals also enhances publisher profits (11). Papers that criticise industry do not generate revenue. Further, many journal editors and editorial board members have financial conflicts of interest with drug firms. The authors describe the manifestation of these problems when submitting papers critiquing Study 329 or CIT-MD-18. They had difficulty finding publication outlets despite the rigour of their work. Some publishers fear legal action from pharmaceutical firms if they publish critical papers (12). Sadly, flawed RCTs seem much easier to publish than articles which critique such work – ironically, this may be especially true when authors present internal drug company documents to buttress their claims. These documents often cast drug companies in a poor light, and this can enhance fear among publishers of involvement in legal action.

Amplifying bias

After research is published in journals, other layers of bias creep in. Pharmaceutical sales representatives spin research findings in a friendly manner in discussions with prescribers. The drug industry pays continuing medical education firms to produce predictably industry-friendly “education” to ensure that prescribers learn about the good news on the latest branded drugs (13, 14). Systematic reviews and meta-analyses synthesise evidence across several RCTs. These reviews frequently ignore the fact that unfavourable findings are sometimes unpublished and that studies are often designed to favour the sponsored product (15, 16). Indeed, overstating treatment benefits and understating risks in published research is standard practice (17, 18, 19). For instance, Turner et al examined the RCTs used for regulatory approval of all 12 antidepressant drugs approved by the Food and Drug Administration (FDA) from 1987 to 2004. Based on the same set of studies, every drug had a higher overall treatment effect reported in journal articles compared to that reported in data submitted to the FDA.

Extracting numbers from journal article reports of RCTs into statistical software can create a meta-analysis, usually without close consideration of problematic study design characteristics or data reporting flaws (16). These meta-analyses form the bedrock of most clinical practice guidelines, which are (without irony) produced and promulgated to ensure that medical providers adhere to the “best evidence”. The EBM Machine churns out RCTs, reviews, and guidelines supposedly based on the Best Evidence, but Jureidini and McHenry point out that we are merely going through the EBM motions without attending to whether the published evidence actually resembles the underlying data.

Key opinion leaders

In many countries, public funding for universities has declined. The drug industry has provided an infusion of money via research funds to study its products. This leads to uncomfortable conflicts of interest, and the authors describe cases of several researchers who raised problems with drugs and found themselves in trouble, sometimes losing their jobs (Chapter 5). In contrast, many academic KOLs have overstated treatment benefits and underplayed treatment risks. They often sign nondisclosure agreements which muzzle them from sharing negative data. In contrast to their whistleblowing peers, KOLs are not penalised. Indeed, KOLs are often quite well-compensated financially. Their industry involvement generates impressive publication lists, bolstering their scientific reputation – regardless of their actual involvement in designing studies, examining the underlying raw data, or writing the papers which bear their name. Indeed, Jureidini and McHenry describe KOLs with well-documented involvement in ghostwritten trials who then became leaders in major medical societies. Universities and medical societies have been largely unwilling to stand up for science, particularly when it may impact their own funding streams from industry.

Not real science

The authors incorporate the work of Karl Popper, whose applications of critical rationalism often appear in The Illusion of EBM (mainly in Chapters 3 and 8). Popper argued that science advances when scientific hypotheses are falsifiable and subject to revision (or discarding) when unsupported by data. Yet industry-funded research is primarily driven by a desire for profit, not the refinement of science. Sometimes, science and the profit motive align. Major pharmacological advances do occur, advancing science and patient well-being. But the authors show that the vast majority of new drugs are “me-too” drugs offering quite similar mechanisms of action to existing competitors. Such drugs usually offer little to no benefit over older, typically less expensive treatments. More germane to Popper’s ideals, scientific theories regarding drug effects are frequently not subject to falsification. When clinical trials fail to support treatment efficacy, data are too often manipulated to make the drug appear effective (19). Theories underlying treatment mechanisms of action are also used as marketing material long after science has failed to confirm their underlying premises, such as the “serotonin dysfunction” theory of depression (20).

Regulatory failure

The authors note that the FDA failed to catch the inappropriate inclusion of unblinded participants in the CIT-MD-18 study. On the basis of one “positive” citalopram trial (CIT-MD-18) and an additional positive (though to an unimpressive degree) study of escitalopram, the FDA approved escitalopram for adolescent depression (Chapter 7). The FDA viewed citalopram as similar enough to escitalopram that a positive trial of citalopram would count toward escitalopram’s efficacy. This contradicts United States federal law indicating that at least two positive trials of a drug are needed for drug approval. Jureidini and McHenry describe comments from former FDA reviewers who were pressured to approve treatments even when RCT data were unsupportive. Having reviewed several FDA reviews of antidepressants for a prior project, I can vouch for the FDA sometimes bending over backwards to allow drugs that do not meet the (relatively minimal) legal standard of two “positive” trials to gain regulatory approval (21). Jureidini and McHenry aptly describe several additional problems in the regulatory process, leading them to conclude that the current process of studying and regulating new drugs does not adequately assess drug efficacy or safety.

There is very little to fault in the authors’ citation of relevant scientific work and internal documents. The Illusion of EBM is an indispensable resource for those interested in EBM and its flaws. This book would be particularly enlightening for those who truly believe in the underlying principles of EBM and are unfamiliar with how industry has largely repurposed EBM to serve the needs of marketing. The perversion of EBM should be widely known by clinicians, researchers, and patients, so that we may gather the necessary momentum to improve how treatments are examined and how RCT data are disseminated.

On that topic, the authors note that several potential fixes have been tried (eg, trial preregistration, legal action against drug firms, reporting of author conflicts of interest, and others) – and that their effects have been minimal (Chapter 8). They propose that industry should not study its own products, especially in controlled trials which may result in regulatory approval. Rather, companies should pay into a fund. Researchers not affiliated with industry would use this money to conduct trials and the data would be a public good, not a private resource that is selectively described in unrealistically positive terms by its sponsor. Perhaps this would work; perhaps industry would find ways around whatever firewalls are erected. In any case, it is difficult to seriously argue that EBM is currently serving its intended purpose. In order to better serve both patients and the pursuit of objective scientific knowledge, the current marketing-oriented implementation of EBM needs major restructuring if not dissolution.