COMMENT

Whither Cochrane?

Meryl Nass, John H Noble Jr

Published online: November 19, 2018


Abstract

The ouster of Professor Peter Gotzsche who headed the Nordic Cochrane Centre, from Cochrane, a respected international research organisation, has provoked a crisis of confidence in the organisation’s future. Disputant and bystander reactions on this issue are presented, as well as concerns regarding conflicts of interest and the reliability of Cochrane reviews. Cochrane’s crisis mirrors the larger crisis of confidence that pervades the entire enterprise of medical research.

We note that within weeks after Gotzsche was expelled from Cochrane, the HPV vaccine (whose Cochrane review he had publicly criticised for conflicts of interest and poor science) received a license expansion in the United States that might be worth billions of dollars to the manufacturer.

Finally, we suggest a variety of new approaches that could strengthen the value of Cochrane analyses, broaden Cochrane’s approach to include additional methodologies, and enhance its independence from financial interests.

Introduction

On July 27, 2018, BMJ’s Evidence Based Medicine published an article by Jorgensen, Gotzsche, and Jefferson (1) criticising Cochrane—formerly Cochrane Collaboration (CC)—for publishing a review (2) of the effect of human papilloma virus (HPV) vaccines on precursors to cervical cancer, in which eligible trials were missed and which breached Cochrane policies and standards. In effect, the HPV vaccine review authors were accused of cherry-picking the data in support of conclusions that were incongruent with the totality of the literature.

On September 17, 2018, the Cochrane Governing Board announced its intention to discipline Peter Gotzsche, a Cochrane founder and member of the Governing Board, for “bad behaviour” (3). Gotzsche’s role as an intrepid critic of corruption in medicine provides context for this decision. First, Gotzsche wrote the provocative book Deadly medicines and organised crime: How Big Pharma has corrupted health care, published in 2013. Second, Gotzsche and Jorgensen (4) had challenged in 2016 the validity of the European Medicines Agency assessment of HPV vaccine safety, highlighting questionable procedural issues, conflicts of interest, and the imposition of lifelong confidentiality agreements.

On September 26, 2018, the Cochrane Governing Board made known its vote from the day before to terminate Professor Gotzsche’s membership and positions in the organisation (5). Revelations by the disputants and bystander reactions to his ouster have shaken confidence in Cochrane, both as a reliable source of integrative research reviews and as an organisation free of conflicts of interest.

The controversy revealed that the conflicts of interest, together with the lack of verifiability of clinical trial data in general that underpins Cochrane’s published meta-analyses, are formidable impediments to the production of reliable research reviews. Yet, there is a dire need for accurate evaluations of drugs, vaccines and medical devices culled from the jungle of published and unpublished biomedical data.

Our frame of reference in assessing the gush of disputant and bystander reactions is the same analytical model demonstrated in Stegenga’s Medical Nihilism (6)—an empirically-grounded, logical demonstration of the essential malleability of all medical research. This includes, in particular, the randomised controlled trial (RCT), whose method is sometimes claimed as the “gold standard” to guide medical decision-making and policy.

Reactions and concerns for the future

Reactions to the Cochrane Governing Board’s statement came fast and furious both from within the organisation and from without. Four of the 12 Governing Board members resigned in protest over Gotzsche’s expulsion from Cochrane, and wrote a letter of explanation, concluding:

It is our hope and deepest desire that this event will encourage all Cochrane members and the wider community to reflect upon where we currently find ourselves and give serious consideration to what we want for the future of Cochrane and its principles, objectives, and ethos. (7)

The reactions were mostly critical of the Governing Board’s decision. They can be sorted into four categories: (a) arguments about “who did what to whom and the wherefores and whys” (8, 9, 10); (b) commentary offering historical perspective while pointing out methodological flaws in Cochrane meta-analytic aggregation of RCT reports (11, 12, 13, 14); (c) explanations appealing to philosophy (15, 16); and (d) commentary focusing on conflicts of interest and need for organisational reform (8, 17). A related source of dispute was Cochrane’s shift from a loose collaboration to centralised management, and its plans for expansion from a current staff of 50 (18) necessitating more ample funding, all of which has been deemed fraught with moral hazard.

Bystander analyses

One bystander, Nass (19), summarised the situation with respect to how the pharmaceutical industry (Pharma) might benefit from Cochrane’s crisis. She further noted that, in the US, a deliberately misleading literature review could subject the authors to a legal charge of scientific misconduct by falsification (20). Her analysis:

  • If the Cochrane HPV review stands, it props the door open for Pharma’s continuing capture of Cochrane’s work and reputation.
  • For Pharma, if Cochrane is captured, it’s a big win.
  • For Pharma, if the (sometimes pesky) Cochrane is not captured, but instead self-destructs, that too is a big win.
  • For Pharma, the only loss would be if Cochrane tightens its belt and its standards, refuses Pharma-laundered funds and refuses to use authors with financial conflicts of interest.
  • Right now, this looks like a position that is unlikely to result, but appears to be the only option to preserve Cochrane as we knew it.

David Healy puts in perspective Cochrane’s longstanding commitment to systematically gathering all RCTs on treatment, winnowing out duplicate reports, and seeking to identify unpublished trials (11)—a good idea, but insufficient to overcome the known problems in the conduct and reporting of RCTs. Healy argues, “If RCTs don’t consistently give the same effectiveness result they are irredeemably flawed.” (12, 13) In an earlier blog, Healy (14) argued that RCTs are “gold standard processors into which we have fed garbage and have got garbage back.”

There is vast literature documenting the influence of corrupt medical research practices, such as concealing adverse events and negative data, penning names to ghost-written publications, publishing biased promotional reports, and hoodwinking the Food and Drug Administration (FDA) by false reporting of efficacy and safety (21, 22). Jefferson and Jorgensen aptly summarise and use the analogy “garbage” to describe these sources of fraudulent and misleading RCT evidence on which Cochrane meta-analytic reviews depend (23).

The real issue is that there are numerous threats to reliability and validity at each of the major stages in the performance of a research review, including at least: (a) problem formulation, (b) identification of relevant studies, (c) judging research quality and controlling researcher and reviewer bias, (d) analysis and interpretation, (e) public presentation and (f) interpreting effect size (24).

Hilda Bastian (9) burrows into the minute details of process and personalities to handle an incendiary situation, seeing it as perhaps “our biggest chance to limit the damage, speed the recovery, and get something positive out of all this.” She discusses the problem of the omitted data identified by Jorgensen, Gotzsche, and Jefferson (1) and the Cochrane chief editors’ claim that it wouldn’t have changed the conclusions of the original Cochrane HPV review (2).

Trish Greenhalgh (15) expresses doubt that Cochrane is experiencing “a crisis of either morality or democracy.” Instead, she asserts, “It’s brand, now as ever, stands for rigour, independence, and a commitment to using science to achieve high-quality patient care and social justice.”

Chair of NoGracias, Abel Novoa (16) argues to the contrary: that Greenhalgh is over-simplistic in couching the decision to expel Gotzsche as a problem separable from organisational governance. Novoa relies on Stegenga’s (6) demonstration that the methodology of evidence synthesis does not produce random results, but disproportionately favours the products evaluated. Novoa points out the critical need for all scientific disciplines to identify and control researcher and organisational bias originating from financial conflicts of interest. A role of Cochrane governance should be to minimise bias in the production of credible research reviews.

BMJ editor-in-chief Fiona Godlee (17) lends support to the position that the Cochrane crisis is much more than tension and clashes between strong personalities, and is, instead, a struggle for the soul of the organisation:

. . . the governing board’s vote to expel one of its founders and most vocal internal critics, Peter Gotzsche, brings to a head years of growing tension between the collaboration’s radical academic roots and its more recent corporate identity … beyond the personalities lies a deep seated difference of opinion about how close to industry is too close. (17)

She expresses the hope that “Cochrane remembers its roots, and that it comes through this episode reinvigorated, independent, and committed to holding industry and academia to account” (17).

In an October 11, 2018 BMJ “Second Opinion” piece about the scandal of vaginal mesh, Godlee (25) again raised the issue of how physicians, researchers and professional bodies are entangled with manufacturers. After so many revelations of how this entanglement worsens the care of patients, Godlee asserts,

We don’t allow judges or journalists to take money from the people they are judging or reporting on. Doctors should be equally independent in their advice to patients… As for industry sponsored research, we welcome the call by Paula Rochon and colleagues for journals to ensure that academic authors retain full control of the process…

Second, given that doctors and researchers do take money from the industry, should the details be readily available to patients and the public? My answer is yes. (25)

What was riding on the Cochrane review of HPV vaccines?

There may have been considerably more riding on the Arbyn et al. Cochrane HPV review (2) than the public knew at the time it came out, or when it was officially disputed by Jorgensen, Gotzsche and Jefferson (1), or subsequently blessed by other Cochrane leaders (26). On October 5, 2018 the FDA raised the upper age limit for Gardasil 9 from 26 to 45 years, more than doubling the number of eligible men and women who could receive it (27). With current worldwide sales over $2 billion yearly (28), this could be a very lucrative approval for Merck.

Moreover, the FDA expanded Gardasil’s market in the absence of data support for either efficacy or safety in the expanded age group.

  • FDA invoked HPV vaccine efficacy by citing only data obtained from women in the older age group who had no prior HPV exposure. In older women whose HPV status was unknown, there was no evidence for efficacy.
  • FDA ignored evidence of increased deaths in women over 25 who had received Gardasil (2).
  • In its approval document expanding Gardasil’s age range, FDA made the outrageous claim that “Solicited AEs (adverse events) were not collected due to the extensive safety database of GARDASIL in younger age groups” (27). The definition of solicited is “sought after”. Is FDA saying that after having sought out AE reports, FDA and the sponsor chose to ignore them?

FDA’s approval was based on sleight of hand, using specious arguments to draw the following conclusions: that injection site reactions were more “substantial” than systemic reactions (or deaths) and, since HPV infection can cause serious disease (despite lack of evidence that HPV vaccine prevents cancers rather than precancerous lesions that may spontaneously normalise), therefore the benefits of the vaccine outweigh the risks in adults aged 26-45. According to the approval document:

Given that the most substantial risks of vaccination with GARDASIL 9 are injection site reactions that are self-limited and mild in severity, the potential benefits of the vaccine, which can prevent serious and/or life-threatening disease, outweigh the potential risks. Therefore, the overall benefit-risk assessment is favorable for use of GARDASIL 9 in adults 27 through 45 years of age. (27)

Had the Cochrane HPV review authors—several of whom had financial conflicts of interest—written a more critical review, or had the Cochrane leaders not officially disputed Jorgensen, Gotzsche and Jefferson’s critique of the Cochrane HPV review, might FDA’s approval of Gardasil’s license expansion have been impeded?

Whither Cochrane?

What can Cochrane do to survive its crisis? We offer suggestions based on disputant and bystander reactions, given increasing revelations of how medical industries have found ways to taint every aspect of the medical research endeavour.

Conflicts of interest

Cochrane must address and fix its conflicts of interest in securing adequate financing from industry and philanthropic and governmental agencies (17). Godlee’s question, “How close to industry is too close?” needs a clear answer now.

Ideally, funding can be found without strings attached, but the harsh reality is, “Whose bread I eat, his song I sing” (German proverb). The Cochrane faces moral hazard and fierce competition in pursuit of adequate sustainable funding with high risk for co-optation and goal displacement regardless of funding source (29). Government funding, which provided a great deal of support to Cochrane, may pose as potent a threat to bias-free independence as industry support, and guidelines for government support need to be as transparent as those for industry.

Advocacy for truth in labelling

Cochrane might reinvent itself as a proactive advocate and lobbyist for truthful and comprehensive reporting of the effectiveness and harms of drugs, vaccines, and medical devices. All data, not only that derived from RCTs, need to be evaluated: clinical study reports, raw trial data, regulatory agency reports on how trials are conducted, evidence from litigation and enforcement agencies, as well as the professional and legal reputations of those engaged in conducting clinical trials. The FDA has in the past licensed drugs using clinical trial data obtained by clinicians who fabricated data—despite several going to prison for doing so (30). The use of disreputable data to determine the proper use of therapeutics can never be acceptable—no matter if it is by Pharma, the FDA, the CDC, European regulators, or Cochrane. Labels describing the proper use of therapeutics must be accurate and comprehensive. Cochrane should exhort agencies such as the FDA, CDC and EMA to use only verifiably accurate data and research methods that do not “spin” data.

Appropriate analytic methods

There is also the matter of using appropriate statistical methods to calculate confidence intervals, perform null hypothesis significance tests, derive p-values, and calculate effect sizes (31). RCT requirements for drug approval do not adequately account for verified prior knowledge—underscoring why so many RCT results are wrong, misleading, and a waste of money (21, 22).

There is need to end the common practices of: (a) comparing a new drug to an inert placebo (or anything less than the current best treatment for efficacy assessment) (b) using subjects in clinical trials who are less likely to show adverse reactions than the population who will receive it (32, 33); and (c) using active, adverse effect-inducing “placebos” (usually novel adjuvants or other vaccines) in vaccine trials, obscuring adverse events caused by the vaccine being tested.

Advocacy for full transparency

Cochrane might reinvent itself as an advocate for full disclosure and sharing among researchers of verifiable data. The goal would be to: (a) promote truth in research reporting (34), (b) minimise opportunities for researcher manipulation of data and statistical analysis to produce biased conclusions, and thus (c) reduce the threats to reproducible science that “…undermine the robustness of published research, and may also impact on the ability of science to self-correct.” (35) There is need to apply existing knowledge (36) to tag, discard or discount, and draw attention to already published, but corrupted, original research reports and any meta-analytic reviews that contain them.

Reinvention as trusted conduit for verified data sharing

More ambitiously, Cochrane might reinvent itself as an independent recipient of verified RCT information via block-chain smart contracting (37, 38). This method permits researchers to transmit to an independent party (such as Cochrane) information about research hypotheses and the details of design and implementation, including data inputs on individual, anonymised RCT subjects. This method enables the identification of changes to data or analytic methods at any step of a trial. The Yale University YODA project (39) provides a template of conditions for allowing researchers, physicians and others who seek access to use clinical trial data.

Conclusion

Cochrane’s reputation for excellence in producing unbiased literature reviews has been seriously tarnished. New scrutiny of the premises underlying the conduct of meta-analyses add to the jeopardy it faces. The likelihood of takeover by industry or government financial largesse, if it has not happened already, seems greater than ever.

Will Cochrane recommit itself to seeking the strongest evidence available to assess therapies, recommit to transparent research methods and governance, and consider flexible research approaches as the problems with individual research methods are identified and understood? Can Cochrane seize this moment to disentangle the processes of medical research from those who have the most to gain by tainting them? Is Cochrane up to the challenge?

References

  1. Jorgensen L, Gotzsche PC, Jefferson T. The Cochrane HPV vaccine review was incomplete and ignored important evidence of bias. BMJ EBM. 10.1136/bmjebm-2018-111012; 27 July 2018.
  2. Arbyn, Xu L. Simoens C, Martin-Hirsch PPL. Prophylactic vaccination against human papillomaviruses to prevent cervical cancer and its precursors. Cochrane Database Syst Rev 2018 May 9;5:CD009068 [cited 2018 Oct 26]. Available from: https://www.cnr-hpv.fr/wp-content/uploads/2018/05/2018-M-ARBYN-COCHRANE-DATABASE.pub3-1.pdf
  3. Statement made by the Governing Board at Cochrane’s 2018 Annual General Meeting, 17th September, at the Edinburgh Cochran Colloquium. [cited 2018 Nov 7]. Available from: https://www.cochrane.org/news/statement-cochranes-governing-board.
  4. Cochrane Nordic. Complaint to the European Medicines Agency (EMA) over maladministration at the EMA. 2016 May 26 [cited 2018 Nov 7]. Available from: http://nordic.cochrane.org/sites/nordic.cochrane.org/files/public/uploads/ResearchHighlights/Complaint-to-EMA-over-EMA.pdf
  5. Statement from Cochrane’s Governing Board – 26th September 2018 [cited 2018 Sep 30]. Available from: https://www.cochrane.org/news/statement-cochranes-governing-board-26th-september-2018
  6. Stegenga J. Medical Nihilism. Oxford: Oxford University Press; 2018. 256 pp.
  7. Gartlehner G, Hammerstein D, Meerpohl J, Santesso N. Letter from the four Board members who resigned in protest over Gotzsche’s expulsion. Deadly medicines and organised crime blogsite. 2018 Sep 15 [cited 2018 Oct 30]. Available from: http://www.deadlymedicines.dk/
  8. Deadly Medicines and Organised Crime blogsite (30 September 2018). Cochrane in moral downfall (ongoing chronology of events relating to PC Gotzsche’s ouster from Cochrane, commentary by Gotzsche and others, and uploaded important documents). 2018 Sep 30 [cited 2018 Sep 30]. Available from: http://www.deadlymedicines.dk/
  9. Bastian H. Boilover: The Cochrane HPV vaccine isn’t really about the evidence – but it’s critical to science. PLOS Blog. 2018 Sep 19 [cited 2018 Sep 30]. Available from: https://blogs.plos.org/absolutely-maybe/2018/09/18/boilover-the-cochrane-hpv-vaccine-fire-isnt-really-about-the-evidence-but-its-critical-to-science/
  10. Just My Thoughts blog. Some thoughts on the Cochrane mess. 2018 Oct 1 [cited 2018 Sep 30]. Available from: https://therealrthorat.video.blog/2018/10/01/the-journey-begins/
  11. Healy D. Cochrane cock-up. Davidhealy.org. 2018 Sep 17 [cited 2018 Sep 30]. Available from: https://davidhealy.org/cochrane-cock-up/
  12. Healy D. Data Based Medicine. Davidhealy.org. 2018 Sep 20 [cited 2018 Sep 30], Available from: https://davidhealy.org/data-based-medicine-and-cochrane-inc/
  13. Healy D. The valley of death. Davidhealy.org. 2018 Sep 25 [cited 2018 Sep 30]. Available from: https://davidhealy.org/the-valley-of-death/
  14. Healy D. The empire of humbug: Not so bad Pharma. Davidhealy.org. 2013 Apr 23 [cited 2018 Sep 30]. Available from: https://davidhealy.org/the-empire-of-humbug-not-so-bad-pharma/
  15. Greenhalgh T. The Cochrane Collaboration—what crisis? BMJ Opinion. 2018 Sep 17 [cited 2018 Sep 30]. Available from: https://blogs.bmj.com/bmj/2018/09/17/trish-greenhalgh-the-cochrane-collaboration-what-crisis/
  16. Novoa A. Greenhalgh is wrong about Cochrane crisis: it’s about science and democracy, not damage control. Nogracias.eu. 2018 Sep 17 [cited 2018 Sep 30]. Available from: http://www.nogracias.eu/2018/09/17/greenhalgh-is-wrong-about-cochrane-crisis-its-about-science-and-democracy-not-damage-control/
  17. Godlee F. Reinvigorating Cochrane. BMJ. 2018 Sep 20 [cited 2018 Sep 30];362:k3966. Available from: https://www.bmj.com/content/362/bmj.k3966.full
  18. Community Cochrane. Implementing Strategy 2020: Cochrane Centres, Branches and Networks. 2015 Nov 30 [cited 2018 Oct 1]. Available from: https://community.cochrane.org/sites/default/files/uploads/inline-files/Centres%20%26%20Branches%20Structure%20%26%20Function%20Review%20paper.pdf
  19. Nass M (18 September 2018). E-mail re: Cochrane Board “Explanation” to Vera H Sharav, cc: Mary Holland, John H Noble Jr.
  20. US Department of Health and Human Services, National Institutes of Health (25 August 2010). Research Integrity. 2010 Aug 25 [cited 2018 Oct 7]. Available from: https://grants.nih.gov/grants/research_integrity/research_misconduct.htm
  21. Ioannidis JPA. Why most published research findings are false. PLOS Medicine. 2005 Aug 30 [cited 2018 Oct 7]. Available from: https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124
  22. Paul Glasziou and Iain Chalmers: Is 85% of health research really “wasted”? BMJ Opinion blog. 2016 Jan 14 [cited 2018 Oct 7]. Available from: https://blogs.bmj.com/bmj/2016/01/14/paul-glasziou-and-iain-chalmers-is-85-of-health-research-really-wasted/
  23. Jefferson T, Jorgensen L. Redefining the ‘E’ in EBM. BMJ Evid Based Med. 2018 Apr [cited 2018 Oct 5].23(2):46-7. Available from: https://ebm.bmj.com/content/23/2/46
  24. Cooper HM. Integrating research: a guide for literature reviews. 2nd edition. Thousand Oaks, CA: Sage Publications;1989. 160 pp.
  25. Godlee, F. What we must learn from mesh. BMJ. 2018 Oct 11;363:k4254.
  26. Tovey D, Soares-Weiser K. Cochrane’s Editor in Chief responds to a BMJ Evidence-Based Medicine article criticizing the Cochrane Review of HPV vaccines. 2018 Sep 3 [cited 2018 Oct 5]. Available from: https://www.cochrane.org/sites/default/files/public/uploads/cochrane_hpv_response_3sep18.pdf
  27. US Food and Drug Administration. Summary basis for regulatory action re Gardasil 9. 2018 Oct 5 [cited 2018 Oct 6]. Available from: https://www.fda.gov/downloads/BiologicsBloodVaccines/Vaccines/ApprovedProducts/UCM622941.pdf
  28. Merck announces fourth-quarter and full-year 2016 Financial Results. 2017 Feb 2 [cited 2018 Oct 5]. Available from: https://investors.merck.com/news/press-release-details/2017/Merck-Announces-Fourth-Quarter-and-Full-Year-2016-Financial-Results/default.aspx
  29. Froelich KA. Diversification of revenue strategies: evolving resource dependence in nonprofit organizations. Nonprofit Volunt Sect Q. 1999; 28(3): 246-68.
  30. Sharav VH. Scientific fraud: Eric Poehlman/Richard Borison. Alliance for Human Research Protection. 2006 Oct 29 [cited 2018 Oct 7]. Available from: http://ahrp.org/scientific-fraud-eric-poehlman-richard-borison/
  31. Cumming G. The New Statistics: Estimation and research integrity. Parts 1-5. Association for Psychological Science. Date unknown [cited 2018 Oct 7]. Available from: https://www.psychologicalscience.org/members/new-statistics
  32. Doshi P. Pandemrix vaccine: Why was the public not told of early warning signs? BMJ. 2018 Sep 20 [cited 2018 Oct 1];362: k3948. Available from: http://bmj.com/cgi/content/full/bmj.k3948?ijkey=Q078zFNInyGN8CS&keytype=ref
  33. Godlee F. A tale of two vaccines. BMJ. 2018 Oct 4 [cited 2018 Oct 7];363:k4152. doi:10.1136/bmj.k4152. Available from: https://www.bmj.com/content/363/bmj.k4152
  34. Carroll BJ, Noble JH Jr, Nardo JM (2016). Stop False Reporting of Drug Benefits and Harms by Making FDA and NIH Work Together. Change.org petition to the US Congress started by the Science and Evidence Counsel of the Right Care Alliance. Available from: https://www.change.org/p/congress-congress-stop-false-reporting-of-drug-benefits-harms-by-making-fda-nih-work-together
  35. Munafo MR, Nosek BA, Bishop, DVM, et al. A manifesto for reproducible science. Nature Human Behav. 2017 Jan 10 [cited 2018 Oct 10];1(1):021. Available from: https://www.nature.com/articles/s41562-016-0021
  36. Noble JH Jr. Meta-analysis: methods, strengths, weaknesses, and political uses. J Lab Clin Med. 2006 Jan;147(1):7-20.
  37. Nugent T, Upton D, Cimpoesu M. Improving data transparency in clinical trials using blockchain smart contracts [version 1; referees: 3 approved]. F1000 Res. 2016 Oct 20; 5: 2541. DOI: 10.12688/f1000research.9756.1.
  38. Will EJ, Noble J. Transparency in scientific publication: time for a new negotiated settlement? J Royal Soc Med. 2018 Sep 18;111(9): 308-9. DOI: 10.1177/0141076818783303.
  39. Yale University Open Data Access (YODA) Project. Date unknown [cited 2018 Oct 29]. Available from: http://yoda.yale.edu/
About the Authors

Meryl Nass ([email protected])

Internal Medicine Physician in private practice, Board Member

Alliance for Human Research Protection, USA

John H Noble Jr ([email protected])

Professor Emeritus

State University of New York at Buffalo, 508, Rio Grande Loop, Georgetown, Texas 78633-4786, USA

Keywords

N/A

Refbacks

There are currently no refbacks.