Published online: February 13, 2017
DOI: https://doi.org/10.20529/IJME.2017.041
We were surprised to read Dr Noble’s article, “Truth in research labelling” (1). Dr Noble quotes from an email exchange he and I had regarding a petition that he had asked the World Association of Medical Editors (WAME) to endorse (personal communication, Bernard Carroll and John Noble, September 27, 2016). Unfortunately, the article’s description of WAME’s comments, which were intended to provide constructive suggestions to improve the petition by ensuring that it was fully supported by facts, is incomplete and the comments have been taken out of context. As Dr Noble notes (but does not quote), WAME indicated, ” …while we agreed with the underlying proposal that the FDA and NIH should ensure that data being reported are consistent, unfortunately the petition includes unsubstantiated statements that WAME cannot endorse. This problem and the fact that it is primarily a US based issue and WAME is an international organization led the Executive to determine that it would not distribute the petition to its members.”
WAME provided five points illustrating examples of unsubstantiated statements. The first point was broken up into two points in the paper, changing the context. Below is an exact copy of the quotes and WAME’s responses in my original email, as sent (shown in italics), including the bolding of the specific language in question that was provided in the original. I have added WAME’s comments regarding Dr Noble’s responses in his IJME article (1).
WAME’s email response: “The problem exists but the frequency is not known.”
WAME’s comment: The three references cited in the original petition (2, 3, 4) were to two essays and a book, not to research studies (which the IJME article now cites). Frequency implies knowledge of numerator and denominator. Several examples of misleading scientific reports have been published, but whether such frequency is high relative to non-misleading scientific reports is not known.
WAME’s email response: “This implies that any existing published pharma-funded studies are unreliable based on ineffective peer-review, which isn’t a known fact and would undermine substantial portions of the medical research literature.”
WAME’s comment: Given that peer-review is the standard that sets scientific research journals apart from other forms of communication, the original statement implies that published industry-sponsored research articles have not met this standard. In fact, when reviewing clinical trials, some journals require submission of research protocols and statistical analysis plans along with the submitted manuscript, and the editors compare the submitted primary outcomes with the primary outcomes originally provided at the time of trial registration, querying authors regarding any discrepancies and requiring clarification of any outcome changes in the revised manuscript. These comprehensive steps can be important to enable effective peer-review in such cases, which is why these journals have implemented such steps.
WAME’s email response: “Most corporate publications” requires the evidence behind this statement. Certainly some, or even many, may, but the idea that most corporate publications were of analyses not specified in the protocol needs more substantial evidence than an assertion.”
WAME’s comment: No citation was provided to substantiate the original claim; Dr Noble has now revised this statement to “many” and provided citations.
WAME’s email response: “This requirement creates the potential for enforcing the creation of inconsistencies between clinicaltrials.gov and the eventual published article, since peer and editorial review can identify inconsistencies and lack of clarity in results reporting that may lead to changes in results prior to publication.”
WAME’s comment: The potential discrepancies between clinicaltrials.gov results and the published trial report could be confusing for readers uncertain as to which version is correct, creating potential issues for clinicians, researchers, and patients. As Dr Noble now notes, it would be necessary to revise the clinicaltrials.gov entry to achieve consistency with the published version. However, it would be the authors who would need to revise the clinicaltrials.gov posting after peer-review and we do not believe this will necessarily be done or be done expeditiously.
WAME’s email response: Again, “regularly fail” is not known from the established instances.”
WAME’s comment: In the petition sent by Dr Carroll on behalf of Dr Noble, no references were provided to support this statement. In the IJME article (1), Dr Noble supports his statement from the previous points, which we disagree with as noted above. He then states, “But more disturbing is evidence that when instructing peer-reviewers to evaluate RCT results, medical journal editors ignore expert opinion about what to look for,” quoting a study (5) which did not reach that conclusion but rather found that peer-reviewers and editors ranked what they considered the most important aspects of reviewing differently (in addition, editors’ requests to reviewers to evaluate the quality of the study was considered by the study authors to be too vague to be included, a decision that likely influenced the result).
WAME had previously indicated to Dr Noble its potential interest in sharing the petition with its members, before the contextual language was added that included the unsubstantiated claims. WAME’s hope was that by providing specific examples of the unsubstantiated claims, it would be possible to modify the document to one supported by existing evidence, which WAME would then have been pleased to consider for distribution to its members and possible endorsement. Indeed, Dr Noble’s email reply to WAME, and his only subsequent communication with WAME, said, “Thanks for reviewing the petition. We will take note of the statements that WAME believes need more substantiation for WAME to endorse. With better substantiation, we hope WAME will reconsider.” Thus we expected that the next step would be the opportunity to review a revised version for possible endorsement. It was a surprise to see the email that WAME provided in the spirit of constructive feedback quoted in this way, without permission or even notification. The published petition (6) is very similar to the version we reviewed, including the original references and language; it does not incorporate Dr Noble’s responses above. (Changes to the published petition include one fewer author and a new title, introduction, and executive summary.)
WAME has great interest in transparency in medical journal publishing, as shown by its policies including the Principles of Transparency and Best Practice in Scholarly Publishing (http://www.wame.org/about/principles-of-transparency-and-best-practice), its policy on handling conflicts of interest of authors, reviewers, and editors (http://www.wame.org/about/conflict-of-interest-in-peer-reviewed-medical), and, most relevant to the issue at hand, Study Design and Ethics (http://www.wame.org/about/recommendations-on-publication-ethics-policie#Study%20Design). The latter policy states: “Good research should be well justified, well planned, and appropriately designed, so that it can properly address the research question. Statistical issues, including power calculations, should be considered early in study design, to avoid futile studies that produce subject risk without enrollment sufficient to answer the research question. Outcomes should be specified at the start of the study. Research should be conducted to high standards of quality control and data analysis. Data and records must be retained and produced for review upon request. Fabrication, falsification, concealment, deceptive reporting, or misrepresentation of data constitute scientific misconduct.” Thus, WAME agrees with the fundamental principles of transparency espoused in the petition (6), but as an association of medical journal editors, we cannot condone making statements without substantiation, just as we would not condone such statements in the journals we edit or have edited.
WAME strongly supports improved transparency in clinical trial reporting, as well as improvements in the editorial process, as noted in its mission statement: “A global association of editors of peer-reviewed medical journals who seek to foster cooperation and communication among editors, improve editorial standards, promote professionalism in medical editing through education, self-criticism, and self-regulation, and encourage research on the principles and practice of medical editing.” We hope that in the future, groups with the same ultimate goals can work together, rather than at cross purposes, to achieve these goals. Ideally, these groups should work to increase the limited resources available to achieve our common goals. The goals are too important and the stakes are too high to set groups one against another.
*The WAME Executive includes Rod Rohrich, MD (President); Christine Laine, MD, MPH (Vice-President); Tom Lang, MA (Treasurer); Lorraine Ferris, PhD, LLM (Immediate Past President); as well as Margaret Winker, MD.