Introduction

On July 27, 2018, BMJ’s Evidence Based Medicine published an article by Jorgensen, Gotzsche, and Jefferson (1) criticising Cochrane—formerly Cochrane Collaboration (CC)—for publishing a review (2) of the effect of human papilloma virus (HPV) vaccines on precursors to cervical cancer, in which eligible trials were missed and which breached Cochrane policies and standards. In effect, the HPV vaccine review authors were accused of cherry-picking the data in support of conclusions that were incongruent with the totality of the literature.

On September 17, 2018, the Cochrane Governing Board announced its intention to discipline Peter Gotzsche, a Cochrane founder and member of the Governing Board, for “bad behaviour” (3). Gotzsche’s role as an intrepid critic of corruption in medicine provides context for this decision. First, Gotzsche wrote the provocative book Deadly medicines and organised crime: How Big Pharma has corrupted health care, published in 2013. Second, Gotzsche and Jorgensen (4) had challenged in 2016 the validity of the European Medicines Agency assessment of HPV vaccine safety, highlighting questionable procedural issues, conflicts of interest, and the imposition of lifelong confidentiality agreements.

On September 26, 2018, the Cochrane Governing Board made known its vote from the day before to terminate Professor Gotzsche’s membership and positions in the organisation (5). Revelations by the disputants and bystander reactions to his ouster have shaken confidence in Cochrane, both as a reliable source of integrative research reviews and as an organisation free of conflicts of interest.

The controversy revealed that the conflicts of interest, together with the lack of verifiability of clinical trial data in general that underpins Cochrane’s published meta-analyses, are formidable impediments to the production of reliable research reviews. Yet, there is a dire need for accurate evaluations of drugs, vaccines and medical devices culled from the jungle of published and unpublished biomedical data.

Our frame of reference in assessing the gush of disputant and bystander reactions is the same analytical model demonstrated in Stegenga’s Medical Nihilism (6)—an empirically-grounded, logical demonstration of the essential malleability of all medical research. This includes, in particular, the randomised controlled trial (RCT), whose method is sometimes claimed as the “gold standard” to guide medical decision-making and policy.

Reactions and concerns for the future

Reactions to the Cochrane Governing Board’s statement came fast and furious both from within the organisation and from without. Four of the 12 Governing Board members resigned in protest over Gotzsche’s expulsion from Cochrane, and wrote a letter of explanation, concluding:

It is our hope and deepest desire that this event will encourage all Cochrane members and the wider community to reflect upon where we currently find ourselves and give serious consideration to what we want for the future of Cochrane and its principles, objectives, and ethos. (7)

The reactions were mostly critical of the Governing Board’s decision. They can be sorted into four categories: (a) arguments about “who did what to whom and the wherefores and whys” (8, 9, 10); (b) commentary offering historical perspective while pointing out methodological flaws in Cochrane meta-analytic aggregation of RCT reports (11, 12, 13, 14); (c) explanations appealing to philosophy (15, 16); and (d) commentary focusing on conflicts of interest and need for organisational reform (8, 17). A related source of dispute was Cochrane’s shift from a loose collaboration to centralised management, and its plans for expansion from a current staff of 50 (18) necessitating more ample funding, all of which has been deemed fraught with moral hazard.

Bystander analyses

One bystander, Nass (19), summarised the situation with respect to how the pharmaceutical industry (Pharma) might benefit from Cochrane’s crisis. She further noted that, in the US, a deliberately misleading literature review could subject the authors to a legal charge of scientific misconduct by falsification (20). Her analysis:

• If the Cochrane HPV review stands, it props the door open for Pharma’s continuing capture of Cochrane’s work and reputation.
• For Pharma, if Cochrane is captured, it’s a big win.
• For Pharma, if the (sometimes pesky) Cochrane is not captured, but instead self-destructs, that too is a big win.
• For Pharma, the only loss would be if Cochrane tightens its belt and its standards, refuses Pharma-laundered funds and refuses to use authors with financial conflicts of interest.
• Right now, this looks like a position that is unlikely to result, but appears to be the only option to preserve Cochrane as we knew it.

David Healy puts in perspective Cochrane’s longstanding commitment to systematically gathering all RCTs on treatment, winnowing out duplicate reports, and seeking to identify unpublished trials (11)—a good idea, but insufficient to overcome the known problems in the conduct and reporting of RCTs. Healy argues, “If RCTs don’t consistently give the same effectiveness result they are irredeemably flawed.” (12, 13) In an earlier blog, Healy (14) argued that RCTs are “gold standard processors into which we have fed garbage and have got garbage back.”

There is vast literature documenting the influence of corrupt medical research practices, such as concealing adverse events and negative data, penning names to ghost-written publications, publishing biased promotional reports, and hoodwinking the Food and Drug Administration (FDA) by false reporting of efficacy and safety (21, 22). Jefferson and Jorgensen aptly summarise and use the analogy “garbage” to describe these sources of fraudulent and misleading RCT evidence on which Cochrane meta-analytic reviews depend (23).

The real issue is that there are numerous threats to reliability and validity at each of the major stages in the performance of a research review, including at least: (a) problem formulation, (b) identification of relevant studies, (c) judging research quality and controlling researcher and reviewer bias, (d) analysis and interpretation, (e) public presentation and (f) interpreting effect size (24).

Hilda Bastian (9) burrows into the minute details of process and personalities to handle an incendiary situation, seeing it as perhaps “our biggest chance to limit the damage, speed the recovery, and get something positive out of all this.” She discusses the problem of the omitted data identified by Jorgensen, Gotzsche, and Jefferson (1) and the Cochrane chief editors’ claim that it wouldn’t have changed the conclusions of the original Cochrane HPV review (2).

Trish Greenhalgh (15) expresses doubt that Cochrane is experiencing “a crisis of either morality or democracy.” Instead, she asserts, “It’s brand, now as ever, stands for rigour, independence, and a commitment to using science to achieve high-quality patient care and social justice.”

Chair of NoGracias, Abel Novoa (16) argues to the contrary: that Greenhalgh is over-simplistic in couching the decision to expel Gotzsche as a problem separable from organisational governance. Novoa relies on Stegenga’s (6) demonstration that the methodology of evidence synthesis does not produce random results, but disproportionately favours the products evaluated. Novoa points out the critical need for all scientific disciplines to identify and control researcher and organisational bias originating from financial conflicts of interest. A role of Cochrane governance should be to minimise bias in the production of credible research reviews.

BMJ editor-in-chief Fiona Godlee (17) lends support to the position that the Cochrane crisis is much more than tension and clashes between strong personalities, and is, instead, a struggle for the soul of the organisation:

. . . the governing board’s vote to expel one of its founders and most vocal internal critics, Peter Gotzsche, brings to a head years of growing tension between the collaboration’s radical academic roots and its more recent corporate identity … beyond the personalities lies a deep seated difference of opinion about how close to industry is too close. (17)

She expresses the hope that “Cochrane remembers its roots, and that it comes through this episode reinvigorated, independent, and committed to holding industry and academia to account” (17).

In an October 11, 2018 BMJ “Second Opinion” piece about the scandal of vaginal mesh, Godlee (25) again raised the issue of how physicians, researchers and professional bodies are entangled with manufacturers. After so many revelations of how this entanglement worsens the care of patients, Godlee asserts,

We don’t allow judges or journalists to take money from the people they are judging or reporting on. Doctors should be equally independent in their advice to patients… As for industry sponsored research, we welcome the call by Paula Rochon and colleagues for journals to ensure that academic authors retain full control of the process…

Second, given that doctors and researchers do take money from the industry, should the details be readily available to patients and the public? My answer is yes. (25)

What was riding on the Cochrane review of HPV vaccines?

There may have been considerably more riding on the Arbyn et al. Cochrane HPV review (2) than the public knew at the time it came out, or when it was officially disputed by Jorgensen, Gotzsche and Jefferson (1), or subsequently blessed by other Cochrane leaders (26). On October 5, 2018 the FDA raised the upper age limit for Gardasil 9 from 26 to 45 years, more than doubling the number of eligible men and women who could receive it (27). With current worldwide sales over $2 billion yearly (28), this could be a very lucrative approval for Merck.

Moreover, the FDA expanded Gardasil’s market in the absence of data support for either efficacy or safety in the expanded age group.

• FDA invoked HPV vaccine efficacy by citing only data obtained from women in the older age group who had no prior HPV exposure. In older women whose HPV status was unknown, there was no evidence for efficacy.
• FDA ignored evidence of increased deaths in women over 25 who had received Gardasil (2).
• In its approval document expanding Gardasil’s age range, FDA made the outrageous claim that “Solicited AEs (adverse events) were not collected due to the extensive safety database of GARDASIL in younger age groups” (27). The definition of solicited is “sought after”. Is FDA saying that after having sought out AE reports, FDA and the sponsor chose to ignore them?

FDA’s approval was based on sleight of hand, using specious arguments to draw the following conclusions: that injection site reactions were more “substantial” than systemic reactions (or deaths) and, since HPV infection can cause serious disease (despite lack of evidence that HPV vaccine prevents cancers rather than precancerous lesions that may spontaneously normalise), therefore the benefits of the vaccine outweigh the risks in adults aged 26-45. According to the approval document:

Given that the most substantial risks of vaccination with GARDASIL 9 are injection site reactions that are self-limited and mild in severity, the potential benefits of the vaccine, which can prevent serious and/or life-threatening disease, outweigh the potential risks. Therefore, the overall benefit-risk assessment is favorable for use of GARDASIL 9 in adults 27 through 45 years of age. (27)

Had the Cochrane HPV review authors—several of whom had financial conflicts of interest—written a more critical review, or had the Cochrane leaders not officially disputed Jorgensen, Gotzsche and Jefferson’s critique of the Cochrane HPV review, might FDA’s approval of Gardasil’s license expansion have been impeded?

Whither Cochrane?

What can Cochrane do to survive its crisis? We offer suggestions based on disputant and bystander reactions, given increasing revelations of how medical industries have found ways to taint every aspect of the medical research endeavour.

Conflicts of interest

Cochrane must address and fix its conflicts of interest in securing adequate financing from industry and philanthropic and governmental agencies (17). Godlee’s question, “How close to industry is too close?” needs a clear answer now.

Ideally, funding can be found without strings attached, but the harsh reality is, “Whose bread I eat, his song I sing” (German proverb). The Cochrane faces moral hazard and fierce competition in pursuit of adequate sustainable funding with high risk for co-optation and goal displacement regardless of funding source (29). Government funding, which provided a great deal of support to Cochrane, may pose as potent a threat to bias-free independence as industry support, and guidelines for government support need to be as transparent as those for industry.

Advocacy for truth in labelling

Cochrane might reinvent itself as a proactive advocate and lobbyist for truthful and comprehensive reporting of the effectiveness and harms of drugs, vaccines, and medical devices. All data, not only that derived from RCTs, need to be evaluated: clinical study reports, raw trial data, regulatory agency reports on how trials are conducted, evidence from litigation and enforcement agencies, as well as the professional and legal reputations of those engaged in conducting clinical trials. The FDA has in the past licensed drugs using clinical trial data obtained by clinicians who fabricated data—despite several going to prison for doing so (30). The use of disreputable data to determine the proper use of therapeutics can never be acceptable—no matter if it is by Pharma, the FDA, the CDC, European regulators, or Cochrane. Labels describing the proper use of therapeutics must be accurate and comprehensive. Cochrane should exhort agencies such as the FDA, CDC and EMA to use only verifiably accurate data and research methods that do not “spin” data.

Appropriate analytic methods

There is also the matter of using appropriate statistical methods to calculate confidence intervals, perform null hypothesis significance tests, derive p-values, and calculate effect sizes (31). RCT requirements for drug approval do not adequately account for verified prior knowledge—underscoring why so many RCT results are wrong, misleading, and a waste of money (21, 22).

There is need to end the common practices of: (a) comparing a new drug to an inert placebo (or anything less than the current best treatment for efficacy assessment) (b) using subjects in clinical trials who are less likely to show adverse reactions than the population who will receive it (32, 33); and (c) using active, adverse effect-inducing “placebos” (usually novel adjuvants or other vaccines) in vaccine trials, obscuring adverse events caused by the vaccine being tested.

Advocacy for full transparency

Cochrane might reinvent itself as an advocate for full disclosure and sharing among researchers of verifiable data. The goal would be to: (a) promote truth in research reporting (34), (b) minimise opportunities for researcher manipulation of data and statistical analysis to produce biased conclusions, and thus (c) reduce the threats to reproducible science that “…undermine the robustness of published research, and may also impact on the ability of science to self-correct.” (35) There is need to apply existing knowledge (36) to tag, discard or discount, and draw attention to already published, but corrupted, original research reports and any meta-analytic reviews that contain them.

Reinvention as trusted conduit for verified data sharing

More ambitiously, Cochrane might reinvent itself as an independent recipient of verified RCT information via block-chain smart contracting (37, 38). This method permits researchers to transmit to an independent party (such as Cochrane) information about research hypotheses and the details of design and implementation, including data inputs on individual, anonymised RCT subjects. This method enables the identification of changes to data or analytic methods at any step of a trial. The Yale University YODA project (39) provides a template of conditions for allowing researchers, physicians and others who seek access to use clinical trial data.

Conclusion

Cochrane’s reputation for excellence in producing unbiased literature reviews has been seriously tarnished. New scrutiny of the premises underlying the conduct of meta-analyses add to the jeopardy it faces. The likelihood of takeover by industry or government financial largesse, if it has not happened already, seems greater than ever.

Will Cochrane recommit itself to seeking the strongest evidence available to assess therapies, recommit to transparent research methods and governance, and consider flexible research approaches as the problems with individual research methods are identified and understood? Can Cochrane seize this moment to disentangle the processes of medical research from those who have the most to gain by tainting them? Is Cochrane up to the challenge?