Abstract

Background: Protocol non-compliance in clinical research studies is common and can affect both patient safety and data integrity. There are no published studies which actively looked for non-compliance. The present study was carried out, against this background, with the objective of assessing the proportion of protocol non-compliance and evaluating those aspects of protocol where there was non-compliance.

Methods: The study completion reports that were submitted to the institutional ethics committee for the period January 2017 to December 2017 were compared with the approved protocol. A checklist for recording protocol non-compliance was developed, which was validated by five experts and consisted of a 12-point checklist with responses such as yes, no, not applicable, and insufficient information.

Results: Out of 193 studies, prospective observational studies were n = 120 (62.17 %), retrospective studies were n = 39 (20.21%), interventional studies n = 28 (14.51 %), and observational studies with both prospective and retrospective study design were n = 6 (3.11%). The study objective was modified in n=18 (9.32%) studies. Only n = 14 (7.24%) satisfied the selection criteria. Six studies (3.10%) did not collect the data as mentioned in the protocol. Fifty-eight studies (30.05%) did not achieve the calculated sample size, whereas n = 78 (40.41%) did not complete the study as per the stipulated study duration. Contrary to 180 protocol deviations found in this study, only 14 protocol deviations were reported by the principal investigator. Aspects like blinding and randomisation, which are relevant to interventional studies (n = 28), showed 100 % compliance.

Conclusion: The research protocol is not adhered to in all aspects. Adequate training to investigators will help prevent non-compliance and enable us to conduct studies with higher ethical and scientific integrity.

Keywords: study design, sample size, interventional studies, non-compliance

Introduction

A protocol is an important document in research that describes the objective(s), design, methodology, statistical considerations, and organisation of a trial that is developed based on evidence-based practice and represents the best method for use of therapeutic regimens (1). Protocols are approved by the institutional ethics committee (IEC) and the regulatory authorities before research studies are initiated. The research team must follow the protocol document religiously if the research project is to comply with all the regulations.

Protocol deviations and violations are the terms used for non-compliance/divergence of a study from the protocol approved by the IEC. Protocol deviation is “non-compliance to protocol approved by IEC which does not affect the safety wellbeing of the participant,” and is also termed a “minor deviation”. Protocol violation is “non-compliance to protocol approved by IEC which affects the safety wellbeing of the participant as well as data integrity of the study” also termed a “major deviation” (2). The US Food and Drug Administration (FDA) defines protocol deviation as “unplanned excursion from the protocol that is not implemented or intended as a systematic change” (3).

Protocol compliance has been a delicate issue in the management of clinical research projects.

Non-compliance with the protocol occurs because the research team may not be adequately oriented and trained to understand its role in the ethical conduct of research or the need to adhere to the regulations; and because the study participants may be poor and uninformed (4). Poor compliance with the protocol may lead to unreliable, misleading, conflicting, and invalid results. In clinical trials, it may reduce the benefit to the research participants or increase the risk of treatment failure. The therapeutic procedures and the drug treatment mentioned in the protocol follow the standard treatment guidelines; if there are some serious deviations, the study may become unscientific and unethical (5).

The role of the ethics committee in identifying protocol non-compliance can be very challenging. It can be carried out passively by reviewing the documents submitted by the investigators to the IEC, which include the protocol deviation form, review form of the continuing study, and study completion reports; or it can be carried out actively by visiting the clinical trial sites (6). Non-compliance and protocol violations are often under-reported by the study team. It is the responsibility of the ethics committee to monitor the working of approved studies for ethical conduct and adherence to the approved protocol (7). No studies that actively looked for protocol noncompliance were reported in the literature. Our study was conducted against this background, to detect whether the observed research studies adhered to the approved protocols with respect to methodology.

Methods

The study protocol was approved by IEC I and IEC II of the Seth GS Medical College and KEM Hospital, respectively.

This was a retrospective observational study.

The study involved evaluating all the clinical projects which were completed during the period January 2017 to December 2017, and the study completion reports for the same period, which had been submitted to the IEC for review. These study completion reports were compared with the approved protocol or protocol amendments to identify non-compliance. The projects which we considered for review comprised of dissertations of post-graduate students, studies other than theses, government-funded research projects, and pharmaceutical industry sponsored studies.

The objectives of this study were:
1. To assess the proportion of protocol non-compliance
2. To evaluate those aspects of the protocol that had not been complied with

We developed a checklist to determine whether the principal investigators had followed the protocol.

The checklist for recording protocol non-compliance was developed using the standard operating procedures (SOPs) of KEM Hospital, the Declaration of Helsinki, 2013, and the Indian Council of Medical Research (ICMR) Guidelines, 2017. This checklist had various points which formed an integral part of the protocol, for example, study objective, study duration, sample size, inclusion and exclusion criteria, data collection tool and technique, mode of treatment, concomitant therapy, efficacy variables, safety reporting, and the statistical tests used (Annexure). Any discrepancies observed in these items were noted in the checklist and ticked as “Yes,” “No,” “Insufficient information,” or “Not applicable.” The content of the checklist was validated by five experts who were members of IEC I and IEC II.

No formal sample size calculations were made for the study. All the study completion reports of clinical projects submitted to the IEC I and IEC II, KEM Hospital, for the period January 2017 to December 2017 were selected. The reason for selecting this period was that the IEC, KEM Hospital, had made the submission of detailed study completion reports mandatory from the year 2016. Thus, it was possible to examine the completion reports against different aspects of the study protocol.

Statistical analysis

The data were presented as percentages and frequency. Descriptive statistics were used to analyse the data. Strict confidentiality was maintained during the data review and analyses.

Results

For the study, the documents reviewed were the study protocol or protocol amendments, whichever were applicable, and the study completion reports. Two investigators reviewed the documents separately. In case of disagreement on any point between these two investigators, the study team members came together for discussion to resolve the conflict.

Type of studies

A total of 193 clinical study completion reports, which were submitted during the period from January 2017 to December 2017, were evaluated. Observational studies formed a majority of the studies, ie 120 (62.17 %), which included both cross-sectional studies as well as prospective observational studies. The retrospective observational studies were the second most common, at 39 (20.21%), followed by interventional studies, ie 28 (14.51 %), and observational studies with both prospective and retrospective study design, ie 6 (3.11%).

Non-compliance with aspects of protocol

Of the total of 193 studies, the study objective was modified in 18 (9.32%) studies, while 112 (58.03%) complied with the study objective, and 63 (32.64%) studies did not give enough information in the study completion report to be able to evaluate whether the objective was modified.

Further, only 14 (7.24%) out of 193 studies modified the selection criteria, and 165 (85.49%) studies followed the inclusion–exclusion criteria strictly as per the study protocol. However, 14 (7.24%) studies gave inadequate information in the report.

Of all the reports, 6 (3.10%) studies did not collect the data as mentioned in the protocol; however, most of the studies (n = 186, 96.37%) did follow the data collection method, and only 1 study (0.5%) did not provide adequate data to confirm whether the data collection tool was followed as per protocol.

With regard to sample size, 135 (69.94%) studies achieved the calculated sample size as mentioned in the protocol. On the other hand, 58 (30.05%) did not achieve the sample size as 40 of the studies had sample sizes less than the calculated sample size stated in the protocol.

We found that 115 (59.58%) studies complied with the study duration as per the study protocol, whereas 78 (40.41%) did not complete the study as per the study duration.

Our study found 180 protocol deviations; while, only 14 protocol deviations were reported by the principal investigator.

As per the other items in the checklist which are specific to interventional studies ─like randomisation, blinding, efficacy variables, safety variables, rescue therapy, and withdrawal criteria ─ all 28 interventional studies ( these aspects are not applicable to observational studies) showed 100 % compliance.

Discussion

The present study, being a retrospective observational study to evaluate compliance with various aspects of the protocol as submitted to the IEC, shows that different aspects in the checklist of the protocol were not adhered to while carrying out the studies.

Protocol deviations may affect scientific integrity or affect the safety or wellbeing of the participants. Hence, the guidelines state that the investigator should promptly report to the ethics committee, the monitor and the sponsor, deviations from or changes in the protocol to eliminate immediate threats to the participants’ safety (8). Those authors who were members of the ethics committee had observed, while reviewing the completion reports, that protocol deviations were not reported by the investigators. Also, while carrying out study site monitoring visits we came across deviations from the protocol which had not been reported by the investigators. This prompted us to search the literature for original studies on protocol deviations.

A thorough literature search revealed original research studies in the area of protocol deviation/non-compliance. However, the methods or study designs used by these researchers to identify protocol non-compliance are different, for example, a review of published clinical trials (9), studies of monitoring reports (10,11), or a review of deviations submitted to IECs (5,12) and so on. A study conducted by Jones et al, who evaluated 45 monitoring reports found that end point deviations (38%) and informed consent document (ICD) deviations (17%) were common (10), and Shetty et al identified ICD-related violations in 8 of the 12 sites monitored by them (11). Whereas a study conducted by Jalgaonkar et al, which evaluated protocol deviation reports submitted to the IEC, reported that of the total deviations reported, the majority were procedure-related deviations (68%) (12). Submission of the completion report for a study is done after completion of the study and is the final stage in the process. We have not found any study actively studying protocol deviations that were not reported by investigators at the final stage of studies. Thus, we planned to study such completion reports to identify the degree of protocol non-compliance.

The study objective states the overall aim of a study. A clearly defined objective directs the researcher to discover answers to questions through the application of scientific procedures. In our study, we found that 9.32% had modified the study objectives. For example, in one case where the study objective was to determine a “correlation” between two variables, it was modified to determine the “association” between the variables. Similarly, in another study, the objective was to study the “prevalence” of a particular outcome, but it was changed to “proportions.” In a few other studies, the number of objectives stated in completion reports were more than the number of objectives stated in the protocol.

In our study, we found that 32.64% of the studies covered did not give adequate information regarding study objectives in the study completion report. The study completion report, according to ICH guidelines, must mention the salient features, which include the study objectives (8). The lack of this component in a large number of submitted completion reports indicate that there is a need for changing practices in the IEC regarding submission of completion reports. It is necessary to ensure that important components of the study are covered in completion reports before they are accepted for review. Also, ethics committee members need to review these completion reports thoroughly and ask for the relevant information from the study team.

Inclusion and exclusion criteria are among the most critical aspects of the protocol. They help to identify the right research participant for the study. Any violation in eligibility criteria in the selection of the right research participants weakens the validity and ethical conduct of the study (13). Similar findings were observed in 7.24% (n=14) of studies, where a few inclusion criteria were not considered while recruiting research participants. In one study, research participants enrolled in the study were beyond the age group mentioned in the protocol. Out of the 14 studies that breached the selection criteria, we found that two studies were interventional studies. As a breach in selection criteria can directly impact the safety of the participants, especially if the study is interventional in nature, we identified the investigators and reprimanded them, making them undergo retraining in Good Clinical Practice. Also, their ongoing interventional studies were tagged for monitoring.

Data collection tools help in achieving the aims and objectives of the research study. We found that the data collection tool stated in the protocol was different from that used in the study completion report in about 3.10 % of the studies. In these studies, the variables evaluated in the protocol and the study completion report were not the same.

Data analysis tools were found to be modified in 6.73% of the studies where statistical tests different from those mentioned in the protocol were applied. For example, tests for association were used as per the completion reports, instead of tests for correlation. Also, statistical tests that were not specified in the protocol were used. For example, the test for regression was specified as a statistical test to be used; however, the same was not applied.

With reference to sample size, we found that the sample sizes stated in the protocol and in the study completion report did not match. Fifty-eight studies (30%) did not achieve the sample size stipulated in the protocol, whereas, there were 40 studies (20%) where the sample size achieved was insufficient. This finding raises serious concerns regarding the scientific validity of the findings as having a smaller sample size than was planned may reduce the power of the study. Such studies are difficult to use for any inference about the reference population. A similar study conducted by An-Wen Chan et al in 1994-95 in Denmark found that only 11 of 62 trials described sample size calculations which were consistent with the protocol and the publication (14).

Out of the 193 studies, 40.41% of the studies did not complete the study within the timelines stipulated in the protocol, when compared with the study completion reports. This finding indicates that the study findings may have reduced relevance, due to an undue increase in the time taken to complete the study.

In addition to all this, when the number of protocol deviations reported to the IEC by the investigators were evaluated, we realised that a meagre 14 protocol deviations were reported voluntarily by the investigators, while 180 deviations were identified in all the 193 studies for which completion reports were submitted in the year 2017.

However, in contrast to the observational studies, compliance with the protocol of interventional studies was satisfactory. These studies were compliant with the study design with respect to randomisation and blinding procedures. This also implies that adequate measures were taken to reduce bias in the study population. The efficacy and safety variables were evaluated as stated in the protocol, and this was reflected in the study completion report. Rescue therapy was followed throughout the study duration as described in the protocol, and so were the withdrawal criteria, as and when need arose during the conduct of the study.

Limitations

This study is limited by the fact that it was a retrospective study and by the fact that it reviewed only the completion reports of completed studies. Had the study been a prospective one wherein studies in progress had been actively monitored, we would have been able to capture non-compliance issues earlier and prevented the patient’s safety being jeopardised. Furthermore, it was not possible to contact each investigator to ascertain the specific reasons for non-compliance with the study protocol. This might have helped develop mitigating strategies for prevention of protocol non-compliance in the future.

Another important task after identifying the protocol non-compliance is the analysis of its impact on patients’ safety and data integrity, a task that has not been undertaken in this study. A study by Ghooi et al categorised protocol deviations into five grades (5). The authors of that study reported that when the protocol deviations were analysed on the basis of their impact, it was noted that the incidence of deviations with minimum impact was high, whereas those with maximum impact were very few. The other studies cited earlier have not only used different methodologies, but in each of them , authors have categorised the observed protocol non-compliance in their own categories, indicating that there is no uniform classification system for protocol deviations. For this reason, it was difficult for us to compare these studies with our study.

Conclusion

This study highlights the need to create awareness amongst study team members about the seriousness of non-compliance with the study protocol and its implications. There is a need to sensitise investigators in their early post graduate training about protocol compliance and good clinical practice. They also need to be trained in seeking IEC approval for protocol amendments and in timely reporting of non-compliance. The ethics committee members need to be extra vigilant and ensure that the study completion reports are submitted as per the guidelines. The IEC members need to improve monitoring and review practices for detecting protocol non-compliance.

Conflict of interest

Dr Sarita Dabba was posted as Senior Medical Officer in the Department of Pharmacology & Therapeutics of Seth GSMC and KEMH while this study was conducted. Dr Sarita was not an IEC member, and was not provided access to identifiable data. Two of the authors who were members of the IECs had declared their conflict of interest and refrained from participating in the review process of their protocol.

Financial support: Self-funded