Introduction

Published medical literature plays a pivotal role in medical training and the teaching and practice of medicine. Day-to-day clinical care, approval for medicines, management guidelines and future research are influenced and guided by the existing medical literature. It has been stated that “the scientific literature is a tangible record of the search for truth”(1).

Published articles can be removed from the medical literature for various reasons. The process of the removal of literature that has already been published is referred to as “retraction”. Retraction has been described as “a statement from the author, institution, editor or publisher stating that an article has significant issues based on unsubstantiated or falsified data or pervasive errors”. (2)

Flawed medical literature can have a deleterious effect on medical science (3, 4). Even retracted literature has been reported to have an impact on published literature, as well as the evolution of science (5). Considering this, it is disturbing to note that the number of retracted articles in medical literature has increased in the past few years (5). According to one report, this number has increased tenfold over the past decade (6).

It is only recently that the issue of retraction in medical literature has begun to be studied systematically, with most of the publications on this theme coming in the past five years (3, 5, 7, 8). It has been hypothesised that the weak research infrastructure in certain countries and the rising pressure to publish in Asia due to theprogress of science inthe continent may have contributed to the increase in retractions (9, 10, 11).

Most of the existing literature on retraction in medical literature is heterogeneous and does not focus on specific specialties. There is no published literature focusing on the retraction of articles in the field of mental disorders. This study aimed to assess the retracted medical literature on mental disorders. It explored the retraction of articles on mental disorders vis-à-vis that of all articles published on the biomedical sciences (henceforth, all articles). It also aimed to test the hypothesis mentioned above on the factors that might be contributing to a greater number of retractions.

Materials and methods

A bibliometric search was carried out, using the PubMed database of the National Center for Biotechnology Information, US National Library of Medicine, USA. The total number of articles published and retracted from the database was ascertained. The Medical Subject Heading (MeSH) terms used for the search for retracted literature on mental disorders were “mental disorders” and “retraction of publication”. All mental and behavioural disorders are listed under the term “mental disorders” and hence, a search using this term yields all publications on mental and behavioural disorders. The search was restricted to human studies in the English language. The analysis covered the literature from 1980 (the year of the first retraction in the literature on mental disorders) till (and including) September 15, 2012. In addition, the Web of Sciences (WoS) database was accessed to find out if the retracted articles were cited even after the publication of the retraction notice.

The article was considered to be “retracted” for the purpose of the present study if it was explicitly retracted or withdrawn via a notice, erratum, corrigendum, editorial note, or other such notification in PubMed (2).

Retraction rates were calculated as the number of articles retracted per 100,000 publications (5). The proportion of all retracted articles to retracted articles on mental disorders was calculated using the total number of publications and publications on mental disorders, respectively. In the case of retracted articles on mental disorders, the type of publication (clinical trial, review article, etc) was ascertained. The retraction notices were retrieved and the reasons for retraction identified.

The ratio of the absolute number of retracted articles to the total number of publications was calculated for each year for all articles as well as for those specifically on mental disorders. Similarly, the ratios of the number of retracted articles to the total number of publications were calculated for all the countries from which articles had been retracted, for all articles on the biomedical sciences as well as for publications specifically on mental disorders.

Earlier research had identified fraud and errors as the reasons for retraction (8). However, there is limited consensus on what should be included under these terms. Fraud refers to instances in which data have been manipulated. It includes fabrication or falsification of data, or a combination thereof. Some of the more recent publications have categorised the reasons for retraction more broadly as different reasons tend to have different implications (5, 12). This was the approach used in our study.

To test the hypothesis that the weak research infrastructure in certain countries might contribute to a higher number of retractions, the countries were divided into high-income countries (HICs) and low- and middle-income countries (LMICs) on the basis of the World Bank classification of countries into different income groups (13). Further, to test the hypothesis that the progress of science in Asia and the attendant pressure to publish might be adding to the problem of scientific misconduct and consequently, retraction, a comparison was made between the Asian countries and the rest of the world.

The data were analysed using SPSS version 21. Between-group analyses were carried out for HICs and LMICs. Similarly, between-group comparisons were made for publications from Asian and non-Asian countries. A non-parametric test (Mann–Whitney U test) was used for this purpose since the absolute number of articles was small. Z-test was used to compare the proportion of retracted articles for different groups. The value of statistical significance was kept at p<0.05 (two-sided) for all the tests.

Results

The search revealed a total of 68 retracted articles. Thirteen of these were excluded (12 were not related to mental and behavioural disorders, while one was not in the English language). Thus, 55 articles were taken up for the final analysis.

The oldest retracted article on mental disorders was published in 1980 (14) and the oldest retraction of an article on mental disorders also took place the same year (14). The 55 retracted articles came from 50 journals. The maximum number of retractions from a single journal was four (15, 16, 17, 18). Three articles had been retracted from one journal (19, 20, 21) and two had been retracted from one (22, 23).

Retraction rates

The retraction rate for all articles (number of retracted articles per 100,000 published articles) indexed in PubMed varied from 2.80 (in 1988) to 30.92 (in 2006) for the period 1980–2012. The retraction rate for articles on mental disorders (number of retracted articles per 100,000 published articles on mental disorders) during the same period varied from 3.56 (in 2005) to 49.25 (in 2012) (Fig. 1).

Fig. 1: Retraction rate for all PubMed articles and articles on Mental Disorders for the period 1980-September, 2013

The retraction rate for all articles remained under 10 per 100,000 published articles till 1999, after which it increased steadily till 2006. Subsequently, there was a decline. Interestingly, the retraction rate for articles on mental disorders has increased steadily since 2005 and is still showing an upward trend.

Countries of origin of retracted articles

A total of 1264 articles on the biomedical sciences in general were retracted from 1980 to 2012. The maximum number of retracted articles were from the USA (313; 39.37%), followed by Germany (151; 18.99%). The maximum number of retracted articles on mental disorders (21; 37.50%) was also from the USA (14, 15, 16, 17, 18, 19, 21, 22, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36). The USA was followed by the UK, which was the origin of 17.85% (10 in number) of the retracted articles. (20, 23, 37, 38, 39, 40, 41, 42, 43, 44). The contribution of the Netherlands (45, 46), Spain (47, 48), China (31, 49), India (50, 51), Iran (52, 53) and Canada (54, 55) to the retracted articles on mental disorders was two each (3.57%). The other countries of origin of retracted articles were Italy (56), Sweden (57), Germany (58), Japan (59), the Czech Republic (60), Singapore (61), Austria (62), Australia (63), Thailand (64) and Pakistan (65) (one article each). The country of origin of two retracted articles could not be ascertained (66, 67).

Types of publication

The types of publication included clinical trials (21%), review articles (20%), case reports/biographies (13%), and abstracts of conference posters (5%).

The retracted articles on mental disorders were most frequently authored by a single person (16 in number; 28.57%) (15, 16, 17, 18, 20, 24, 25, 30, 31, 54, 55, 64, 67, 68). Ten articles (17.85%) were written by three authors (27, 38, 40, 47, 50, 51, 52, 60, 62, 65), while eight were (14.28%) written by four (19, 22, 29, 33, 39, 41, 53, 56).

Reasons for retractions

Of the 38 articles for which the retraction notices could be accessed, nine (24.32%) were retracted for fraud. The most common type of fraud was falsification of data (7 articles) (20, 33, 41, 46, 49, 57, 58). One article was retracted for fabrication of data (54), and one for both falsification and fabrication of data (28). Plagiarism was cited as the reason for retraction in the case of five (13.51%) articles (24, 30, 50, 51, 62). Seven (18.91%) articles were retracted because they were duplicate publications (26, 32, 45, 48, 52, 56, 68). One article was retracted erroneously and reinstated later on (37). Table 2 lists the various types of errors cited as reasons for retraction. Eight (14.54%) of the 55 retraction notices did not cite the reason for retraction (23, 39, 47, 53, 59, 60, 66, 67).

Proportion of articles retracted

Of all articles on the biomedical sciences, 0.0138% had been retracted. The corresponding percentage for articles on mental disorders was 0.0095%. The proportion of retracted articles was significantly higher for all articles as compared to the articles on mental disorders (z=2.87, p=0.004).

The countries that contributed at least one retracted article on mental disorders were analysed further. It was found that in the case of these countries, 0.0196% of all articles and 0.0172% of articles on mental disorders had been retracted. The proportion of retractions for all articles and articles on mental disorders was comparable (z=0.89, p=0.37).

During the period 1980–2012, of the 255,589 articles on the biomedical sciences originating in the LMICs, 83 were retracted. The corresponding figure for the 8044 articles on mental disorders was eight. During the same period, 759 of the 4,050,922 articles on all biomedical sciences originating in the HICs were retracted. The corresponding figure for the 299,482 articles on mental disorders was 48. In the case of LMICs, 0.032% of all publications and 0.099% of publications on mental disorders were retracted. The corresponding figures for HICs were 0.018% and 0.014%, respectively.

Fig 2 (a). Proportion of articles retracted for PubMed publications for individual countries for the period 1980-September, 2012 (the red line represents the median value)

Fig 2 (b). Proportion of articles retracted for PubMed publications on Mental Disorders for individual countries for the period 1980-September, 2012 (the red line represents the median value)

The median value for the percentage of retracted articles on all biomedical sciences was 0.0173% (range – 0.009% [for the Netherlands] to 0.106% [for Iran]) (Fig. 2(a)). The median value for the percentage of retracted articles on mental disorders was 0.0296 (range – 0.006% [for Germany] to 0.495% [for Pakistan]) (Fig. 2(b)). The percentage of retracted publications in the case of all articles was greater than the median value for all Asian countries, Austria and Germany. Similarly, the percentage of retracted articles on mental disorders was greater than the median value for all Asian countries, Austria, the Czech Republic and Spain.

Inter-country differences in retraction

The HICs contributed a significantly greater number of publications on all biomedical sciences than the LMICs (U=10.00, p=0.027). They also contributed a significantly greater number of publications on mental disorders (U=8.00, p=0.016). No significant differences were observed between the HICs and LMICs with respect to the absolute number of retraction of articles on all biomedical sciences (U=24.50, p=0.429), absolute number of retraction of articles on mental disorders (U=28.50, p=0.658), absolute number of retraction of articles on mental disorders due to fraud (U=26.00, p=0.462), and absolute number of retraction of articles on mental disorders due to other reasons (U=28.50, p=0.667). However, compared to the HICs, a disproportionately greater number of articles was retracted in the case of publications originating in the LMICs. The difference was significant both for all articles (U=3.00, p=0.004) and articles on mental disorders (U=5.00, p=0.007).

Also, a disproportionately greater number of articles was retracted from Asian countries compared to non-Asian countries. The difference was significant both for all articles (U=12.00, p=0.016) and articles on mental disorders (U=4.00, p= 0.002).

Citation of retracted articles

Interestingly, 20% of the articles were self-cited by the authors and 7% were self-cited even after retraction notices had been published. Around 39% of the articles had been cited at least once in the WoS, even after they had been retracted from PubMed.

Discussion

The database used for this study was PubMed. PubMed is administered by the National Library of Medicine in the USA and contains nearly 25 million records (12). It is a commonly used database for medical literature the world over. It offers free search access through its interface and its coverage of the biomedical literature is comprehensive (69). Search of the PubMed database revealed that the first retracted article on mental disorders was published in the year 1980- the same year that the first article on mental disorders was retracted from PubMed.

Not many attempts have been made to systematically study the retraction of biomedical articles. Some of the notable studies include those by Steen (8), Fang et al (12) and Griensein et al (12). Others have studied this issue in more restricted ways. For example, Foo (70) included studies from authors whose articles had been retracted more than thrice. To date, no study has explored the retracted literature on mental disorders in detail.

A total of 55 articles on mental disorders were retracted over a period of 22 years. In the case of three journals, there was more than one retraction. The maximum number of retractions from a single journal was four. Other studies have reported up to 123 retractions from a single journal (5). However, our study found that fewer articles are retracted from the majority of journals (5). The earliest retracted article was published in 1973 and retracted in 1977 (12). The first retracted article on mental health was published in 1980 and this was also the year when the first article on mental health was retracted.

Research has shown an increase in the number of retracted articles over the past few years (5, 8, 12). In our study, it was observed that the retraction rate for all articles peaked in 2006 and has shown a steady decline since. However, the retraction rate for articles on mental disorders is still showing an upward trend.

The maximum number of articles on all biomedical sciences (around 39%) was retracted from the USA. The same was true of articles on mental disorders. Previous studies have found that the USA is the most common country of origin of retracted articles (5, 71). Steen found that around 33% of all retracted articles were from the USA (71). There has been a change in the trend since 2005, with an increase in the number of retracted papers originating from China (5).

However, it is important to look at the absolute number of retractions from an individual country in the light of the country’s contribution to the pool of published literature. While the USA accounts for the maximum number of retracted articles (all articles as well as those on mental disorders), it also contributes a disproportionately greater number of articles than other countries. In fact, if we consider the percentage of retracted articles as a ratio of the total publications from a country, we find that the USA accounts for a rather low percentage of the retracted articles. The percentage of articles retracted from all articles was found to be the highest for Iran (0.106%). The percentage of articles retracted from articles on mental disorders was the highest for Pakistan (0.495%). In fact, the percentages of articles retracted from all articles as well as from articles on mental disorders were greater than the median value for all Asian countries. Among the non-Asian countries, the values were greater than the median for Austria (both for retracted articles on all biomedical sciences and on mental disorders), the Czech Republic (for retracted articles on mental disorders), Germany (for retracted articles on all biomedical sciences) and Spain (retracted articles on mental disorders).

It has been hypothesised that countries with poor research infrastructure make a relatively greater contribution to retracted literature (71). Also, it has been suggested that the progress of science in Asian countries, with the attendant pressure topublish, might lead to an increase in scientific misconduct (10). Concerns have been expressed about the increasing instances of scientific fraud in some Asian countries, such as China (9). However, Steen has dismissed the possibility that China is a disproportionately larger contributor to retracted literature (71). It is important to note that the observation regarding China was based on the absolute number of publications. If we look at the proportion of retracted articles vis-à-vis the contribution to the pool of published articles, all Asian countries tend to score above the median value for retracted articles on all biomedical sciences as well as on mental disorders. This is not an exclusively Asian phenomenon as the percentages for certain non-Asian countries, such as Austria, the Czech Republic, Germany and Spain, are also greater than the median values. Also, it has been found that the recent peaks in the retraction of articles from some Asian countries are as a result of an increase in the retraction of articles by repeat offenders. For example, in the study by Grieneisen et al (5), the exclusion of articles by two authors made the retraction rate for China drop to US levels for the year 2006.

Similarly, the argument that countries with poor research infrastructure are likely to make a relatively greater contribution to retracted literature is not entirely correct. Certain developed Asian countries, such as Japan and Singapore, and developed non-Asian countries also make a disproportionately large contribution to the pool of retracted literature. Overall, however, LMICs tend to make a disproportionately larger contribution to the pool of retracted articles on all biomedical sciences as well as those on mental disorders. Other studies have also reported an increase in the number of retracted articles from Asian countries since 2005 (5).

Our study found that fraud was the most commonly cited reason for retraction. A previous study assessing multiple areas of publication, including the biomedical sciences, had found that 16.6% of the retracted articles had been withdrawn due to fabrication and falsification of data (5). “Publishing misconduct”, including plagiarism, duplicate publication, authorship issues and copyright issues, had been the cause of 46% of the retractions. The reasons for retraction had not been specified in 14% of the retraction notices. In our study, “publishing misconduct” accounted for 57.87% of the retractions. No reasons were cited in 17.39% of the retraction notices. In the study by Steen (8), fraud accounted for 28% of the retractions. The reason for retraction was not specified in 8% of the retraction notices. The study by Fang et al (12) found that 49% of the retractions were due to fraud. Twenty-four per cent were due to errors and 11% due to plagiarism. The COPE survey of Medline retractions from 1988 to 2004 reported that 28% of the retractions were due to research misconduct (including fabrication or falsification of data) (72). However, the study by van Noorden (6) reported that fraud was responsible for only 11% of the retractions. This study reported that plagiarism (including self-plagiarism) was responsible for 33% of the retractions. The variation in rates across different studies can be explained by the use of different inclusion criteria and databases.

According to our study, only 0.0138% of all articles and 0.0095% of articles on mental disorders were retracted from 1980 to 2012. According to another study, the proportion of retractions of publications on “psychiatry” was lower, at 0.0017 (5). The same study found that the proportions of articles retracted in the categories of “medicine, general and internal” and “medicine, research and experimental” were also lower, at 0.0026 and 0.0054, respectively. The database, search strategy and search terms used in this study were different from those used in our study. As in our study, the proportions were greater (even more than two times for some years) for PubMed-based analysis than WoS-based analysis.

It can be argued that the percentage of articles retracted is too low to have any potentially relevant impact of any kind. However, it has been hypothesised that the retractions which have been detected might merely be the tip of the iceberg as many more cases of research misconduct and errors remain hidden (73). According to Cokol et al (74), on the basis of their model, at least 10,000 of the articles published between 1950 and 2004 ought to be retracted.

Many retracted articles continue to be cited as valid research (75, 76, 77, 78). Moreover, highly cited articles continue to be cited at a greater frequency even after retraction (78). Steen (79) reported that retracted articles published in high-impact factor journals continue to be cited on an average of 18.4 times and those published in low-impact factor journals continue to be cited on an average of 3.6 times even after retraction. Articles are also self-cited even after they have been retracted by the author. Our study found that 20% of articles were self-cited by the authors and 7% were self-cited even after retraction notices had been published. Around 39% of the articles were cited at least once in the WoS even after retraction from PubMed. Grieneisen and Zhang (5) reported that 1837 items which were published in the WoS and marked “retracted article” (beside the title) were cited 41, 562 times. However, the authors did not specify whether these citations excluded those that criticised the retracted articles.

Clinical trials constituted 21% of the retracted articles on mental disorders in our study. In Steen’s study (8), 13% of the retracted articles were clinical trials. Clinical trials are relatively overrepresented in retracted articles. This is so because only around 6% of published articles are randomised controlled trials (8). Clinical trials are of specific relevance in the context of retracted literature as flawed studies can put the participants at risk. While the original studies might include a small number of subjects, secondary studies based on these studies recruit a much larger number of subjects. One study analysing the issue of participants being “at risk” as a result of retracted literature found that while 28,783 subjects were enrolled in studies that were eventually retracted, approximately 445,064 were enrolled in secondary studies citing a retracted primary paper (3). Of these, 42.6% received some active intervention. While the retracted studies exposed a lower number of participants to flawed research, the secondary studies based on these retracted studies exposed a much higher number to flawed research.

Previous studies have reported that having many authors does not prevent research fraud (71). In our study, one retracted article was written by 13 authors. However, around 29% of the retracted articles were written by a single author, which was greater than the proportion of single-author articles reported by Steen (7.23%) (71). Another study that included articles by authors who had retracted more than three articles found that only 6.6% of the articles had a single author (70).

Retraction in medical literature has reportedly increased around tenfold in the past decade, even though the number of published articles has risen by only 44% during this period (6). The retracted literature has been reported to have had an impact on the literature published, as well as the evolution of science (5). Retracted literature is an indicator of the failure of scientific process. It represents wasted resources, diverts the search for truth, wrongly influences clinical practice and research, and can erode public confidence in science.

It has also been hypothesised that open access to articles facilitates the detection of misconduct in research and publication and hence, expedites the process of retraction. This hypothesis has been supported by findings from a recent study on retracted articles on mental disorders (80). A significantly greater number of articles cited in the WoS following retraction were non-open access in PubMed Central in this study. However, open access did not influence the amount of time taken to retract an article following its publication.

Our study has made some important findings. The retraction rate for articles on mental disorders, although lower than that for all articles, is still on the rise. While the retraction rate for the latter peaked in 2006, it is still showing an upward trend for articles on mental disorders. Articles have been retracted from 50 different journals. As opposed to biomedical literature in general, the largest proportion of retracted articles on mental disorders tends to be written by single authors.

Although HICs such as the USA, the UK and Germany contribute the maximum number of retracted articles, they are, at the same time, disproportionately bigger contributors to the pool of articles on all biomedical sciences. The proportion of articles retracted is greater in the case of publications originating in the LMICs than the HICs. However, it is not correct to make general statements such as the Asian countries or countries with poor research infrastructure are likely to be bigger contributors to the retracted literature. There are exceptions to both these comments. However, all Asian countries do have a higher retraction rate for articles on mental disorders than for all articles. Research fraud is the commonest reason cited for the retraction of articles on mental disorders. It is important to address the various reasons behind these frauds and errors. Researchers must be educated on these issues. Further, journals must make use of the latest technologies to deal with research misconduct.

Interestingly, this was the first study to explore the retracted literature on mental disorders. PubMed was used as the database for the study. Although PubMed is a commonly used database for searching the biomedical literature, it is much smaller than certain other databases (71). Hence, the study did not include some of the relevant literature. The study screened all publication types across all countries. However, the search was restricted to human studies in the English language. Future studies that include literature in languages other than English and employ other databases would help us gain a more comprehensive understanding of the situation.

Conflicts of interest: None

Financial disclosures: None

Note

On November 6, 2014, the sentence “It was in 1980 that the PubMed database was first searched for published literature – the same year that the first article on mental disorders was retracted from PubMed.” was corrected as follows”. “Search of the PubMed database revealed that the first retracted article on mental disorders was published in the year 1980- the same year that the first article on mental disorders was retracted from PubMed.”