A simulated trial with reinforcement learning for the efficacy of Irinotecan and Ifosfamide versus Topotecan in relapsed, extensive stage small cell lung cancer

Abstract

Objectives

Synthetic data may proxy clinical data. At the absence of direct clinical data, this study aimed to compare Irinotecan and Ifosfamide (II) with Topotecan in synthetic, recurrent small cell lung cancer (SCLC) patients within a simulated clinical trial.

Materials and methods

Two simulation stages were conducted. Initially, 200 recurrent SCLC cases were simulated to replicate a previous phase 3 trial, testing the utility of Cox proportional hazards model and simulation methodology together, where patients were randomized to receive Cyclophosphamide, Adriamycin, Vincristine (CAV) or Topotecan. In the second stage, 600 recurrent SCLC patients were simulated and randomized to compare Topotecan versus II in terms of overall survival (OAS), using Reinforcement Learning (RL) and Cox proportional hazards model.

Results

CAV versus Topotecan comparison showed no statistical difference in overall survival (hazard ratio (HR): 0.89, 95% CI: 0.67–1.18, P = 0.418), aligning with the original clinical trial. For the Topotecan versus II comparison, the RL framework significantly favored the II arm (mean reward points: 193.43 versus − 251.82, permutation P < 0.0001). Likewise, II arm exhibited superior median OAS compared to Topotecan arm (11.12 versus 6.30 months). HR was 0.44 (95% CI: 0.38–0.52) with P < 0.0001, in favor of II.

Conclusion

Artificial trial results for CAV versus Topotecan matched the original trial, confirming indifference of OAS. Additionally, II yielded superior overall survival compared to Topotecan in recurrent SCLC patients. These demonstrate the potential of RL and simulation in conjunction with Cox modelling for similar studies. However, definitive conclusions necessitate a randomized clinical trial between II and Topotecan in this patient cohort.

Introduction

Extensive stage SCLC has a dismal prognosis at the time of initial diagnosis, with an expected survival of 7 to 11 months with combination chemotherapy, and of only 1.5 months without chemotherapy, and this negative prognostic outlook has not changed much over the course of last 30 years [1]. Only recently, the addition of immunotherapy like Atezolizumab or Adebrelimab to the first line treatment in the metastatic setting provided a marginal overall survival improvement to a total of 12 to 15 months [2, 3]. Likewise, when there is disease progression, the prognosis is again dismal; the expected median overall survival with the second line Topotecan, the current standard, is only 5.8 months (25 weeks), and the improvement in quality of life is more pronounced with Topotecan as compared to CAV [4].

The activity of Irinotecan Ifosfamide (II) combination was firstly reported in recurrent or progressive SCLC in 2003, by Ichiki, et al. [5]. Our group also confirmed the activity of II in the same setting and we observed a median overall survival (OAS) figure of 11.1 months with the II regimen [6]. However, currently there is no report of a randomized controlled trial that compares the efficacy of II combination with the Topotecan regimen in relapsed or newly diagnosed patients.

Reinforcement Learning (RL) is a type of machine learning, like supervised and unsupervised learning methods [7]. Its famous application in the Go game, and the success of the resultant artificial intelligence model Alpha Go to beat a 9-dan professional in 2016 demonstrated the feasibility and the efficacy of RL in gaming [8]. With the usage of “rewards” and “punishments”, a virtual agent interacts with an environment to choose the best mode of action towards a goal. Utilization of RL in clinical medicine in general, and in oncology in particular, has been quite new, and not been tested thoroughly. Of note, some recent publications attempted to find the best dosage of a chemotherapy drug or the most suitable radiation treatment planning for individual patients, and optimize dynamic treatment regimens over time [9,10,11].

Patient simulation has been employed as a technique to evaluate treatment regimens and inform decision-making in artificial clinical trials. Through the development of computational models and virtual patient populations, researchers have gained the ability in the field of oncology, to optimize therapeutic approaches, predict outcomes, or improve clinical trials [12, 13]. These advancements hold promise for accelerating the development of effective cancer treatments and improving patient care.

In this study, thus, we made use of RL methodology, aiming to compare the activity of II with that of Topotecan, in terms of overall survival, in patients with extensive SCLC and who relapse or progress after 1st line chemotherapy, using completely simulated patients. Prior to this, we also tested the simulation technique we used in this paper by reconducting a previously published clinical trial that compared CAV versus Topotecan, this time as an artificial trial, to test the validity of our methodology for conducting an artificial phase 3 trial [4].

Materials and methods

The whole project was completed in Python coding language, and a number of Python libraries were included, namely; numpy, pandas, scikit-learn, matplotlib, seaborn, and lifelines. Google Collab was used as the coding medium [14].

The methodologies used in patient simulation, randomization, and Cox modelling are identical both for the initial stage in testing CAV versus Topotecan, and second stage in comparing II versus Topotecan. RL was used for the II versus Topotecan comparison, for the second stage, in conjunction with Cox modelling. These methodologies are described below. During the preparation of this work, we used Chat GPT 4.0 in order to improve readability in some parts of the manuscript. After using this tool, we reviewed and edited the content as needed. Consent to Participate was not obtained as no human subjects were involved. Likewise, no ethics declaration is made in this paper because this is a simulation trial. We did not receive any funding for this study.

Patient simulation, randomization

In this study, patient simulation was employed to create a virtual patient population for evaluating treatment efficacy in the context of an artificial clinical trial. A total of 600 patients with recurrent SCLC were simulated for the main comparison; II versus Topotecan (second stage of the study), and 200 cases were simulated for CAV versus Topotecan (initial stage of the study), incorporating factors such as gender, age, and Eastern Cooperative Oncology Group (ECOG) status, with distributions as reported in the literature for this group of patients [4, 6, 15, 16]. Modifications were applied to the survival rates based on age and ECOG status, mimicking real-world scenarios. An age profile above 70 as an indicator of advanced age was used to modify the associated OAS rate by 10% reduction, because advanced age was shown to be a probable adverse prognostic factor in SCLC, and patients received chemotherapy less frequently with the advancing age [17]. Likewise, an ECOG status of 2 prompted a 55% reduction in OAS in our study, since in real life, a negative correlation of an ECOG performance status of 2 or higher with OAS in SCLC patients has been reliably established [17]. The patient data, including gender, age, ECOG status, and simulated survival durations, were stored in a data frame for further analysis. By randomizing the assignment of patients to treatment arms with the use of NumPy Python library, and generating survival durations using the Weibull distribution, the study allowed for comparisons between the two treatment arms. The employment of Weibull distribution and mean survival times have been utilized in the literature to compare treatment effects in oncology [18, 19].

Reinforcement learning methodology

The methodology employed in this study leveraged reinforcement learning principles to optimize treatment decision-making within the simulated patient population. The study aimed to identify the treatment arm that would yield the highest rewards based on survival outcomes. A reward and punishment system was established, where rewards were assigned for hypothetically extending patient survival by two months, and punishments were given for survival decreases of the same duration. OAS extension by 2 months has previously been recognized as a clinically meaningful endpoint for cancer patients with metastatic disease, and we chose the same threshold for reward and punishment definition in this study [20]. In this paper, we used the RL methodology to compare II and Topotecan arms. Through a series of iterations, the patient data were shuffled and split into two treatment arms. As a starting point, baseline OAS figures were chosen from the literature as; 5.8 months for CAV, 6.3 months for Topotecan, and 11.1 months for II [6, 21, 22]. The baseline OAS figure for CAV was used in initial stage of the study (CAV versus Topotecan comparison) for simulation and Cox analyses, whereas, the baseline OAS figures for Topotecan and II were used in the simulations, RL and Cox analyses, during the second stage of the study. Particularly, median OAS values from 4 phase 2 studies of Topotecan in this setting were averaged and rounded, and the result of 27 weeks is expressed as 6.3 months as the baseline Topotecan OAS figure. [23, 24]. Random values from the Weibull distribution were generated and added to the baseline OAS figures to simulate survival durations for each arm. The total rewards for each arm (II versus Topotecan) were computed, and statistical tests were conducted to assess the significance of differences in mean rewards between the two arms. Number of iterations in the RL model was 1000. Additionally, a permutation test was performed to validate the observed test statistic. A p value of less than 0.05 was considered as significant.

The RL methodology, while primarily used here to guide a straightforward treatment decision, serves as a complementary approach to the Cox model. When used together, the two methods provide a robustness check, ensuring that both are capable of identifying the superior treatment option for II and Topotecan arms. By linking the RL-based reward comparison between the II and Topotecan arms with the hazard ratios estimated by the Cox model for these treatments, we enhance the robustness of our conclusions and ensure that both methodologies consistently indicate similar treatment effects. This dual approach helps reinforce confidence in selecting the more effective treatment arm. In conducting RL, we utilized an offline learning approach with a finite time horizon, in the context of a basic policy evaluation.

Cox model

In addition to RL to test the OAS difference between the II and Topotecan study arms, in this study, the Cox proportional hazards regression model was employed to examine the impact of treatment interventions on patient survival for CAV versus Topotecan, and II versus Topotecan comparisons. The survival data, comprising the combined survival durations and treatment arm labels, were incorporated into the model. By fitting the Cox model, the hazard ratio was estimated, providing insights into the relative risk of survival between the treatment arms. Confidence intervals were computed to assess the precision of the hazard ratio estimate, and a p-value was obtained to evaluate the statistical significance of the treatment effect. The utilization of the Cox model allowed for the identification of potential associations between treatment arm and patient survival, accounting for the varying risks over time. This analysis provided information regarding the relative effectiveness of the treatment interventions in influencing patient outcomes and furthered the understanding of the impact of the 2 treatments (II versus Topotecan) on survival in the simulated patient population. Again, a P value below 0.05 was regarded as statistically significant.

In this study, survival times for each patient were modeled using the Weibull distribution, which is widely used in oncology for its flexibility in representing time-to-event data. The survival times generated were adjusted based on patient age and ECOG status to reflect real-world survival modifications, such as a 10% reduction in survival for patients over 70 and a 55% reduction for ECOG 2 patients, as pointed out before. Thus, we used Cox proportional hazards modeling with adjustments for confounders such as age and ECOG status in both the II versus Topotecan and CAV versus Topotecan comparisons. Additionally, the simulation of the II versus Topotecan study arms was designed to ensure similar distributions of these confounders, allowing for a more accurate evaluation using reinforcement learning (RL). These steps were taken to better reflect the causal relationship between the treatment arms and the observed improvements in survival.

The treatment effects in this study were modeled under the assumption that extending overall survival (OAS) by 2 months is a clinically meaningful endpoint, and this extension was used as the threshold for reward in the reinforcement learning framework. Baseline survival times for the control arm (CAV: 5.8 months), Topotecan (6.3 months), and II (11.1 months) were derived from previously published literature. Randomized allocation to treatment arms was assumed to be unbiased, and the Cox model accounted for time-varying risks. The model assumed proportional hazards across treatment arms, meaning that the relative treatment effect (hazard ratio) remains constant over time. Additionally, no cross-over between treatment arms was allowed, and no additional interventions were simulated post-treatment assignment.

Results

General features

For the initial stage to test CAV versus Topotecan by the Cox model, 200 cases were simulated and randomized, and patient features were well balanced as in accordance with the original phase 3 study⁴. The main characteristics are evenly balanced after the simulation process, such as the male gender distribution is 91% versus 90% in CAV and Topotecan arms, respectively. Refer to Table 1 for details.

Table 1 Patient characteristics for Topotecan and CAV comparison

For the second stage of the study, in order to test II versus Topotecan both by RL and Cox model, a total of 600 SCLC cases with progressive disease were randomized into II and Topotecan arms, with 300 cases in each treatment group. Due to the simulation and randomization process, the distribution of the patient features was similar with a mean age of around 50, male predominance, and the majority of the cases had an ECOG performance score of 0 or 1. Similar to first stage of the study, where CAV and Topotecan arms were compared, the simulation of II versus Topotecan arms captured the equivalence of distributions of important clinical characteristics across the study arms. See the details in Table 2. In this direction, Figs. 1, 2 and 3 highlight the similarity of age, gender and ECOG performance status distributions, respectively, in II and Topotecan arms.

Table 2 Patient characteristics in Topotecan and II arms

Fig. 1

Age distribution for the Topotecan and Irinotecan Ifosfamide arms

Fig. 2

Gender distribution for the Topotecan and Irinotecan Ifosfamide arms

Fig. 3

ECOG performance score distribution for the Topotecan and Irinotecan Ifosfamide arms

Reinforcement learning results

When II and Topotecan were compared in efficacy, the mean reward figures for the Topotecan and II arms were − 251.82 and 193.43, respectively, and the difference was 445.25 in favor of the II arm (Permutation P value < 0.0001). Likewise, the difference was statistically significant with other tests; T statistic = -985.45, T test P value < 0.0001, and U-Statistic: 0.00, Mann-Whitney U test P value < 0.0001. So, the II arm had more rewards after simulation, by the reinforcement model, with a statistical significance. See Fig. 4 for the distribution of reward points across the study arms of Topotecan and II. Figure 4 clearly illustrates the separation of reward distributions for the Topotecan and II arms.

Fig. 4

Reinforcement learning reward points for the Irinotecan Ifosfamide and Topotecan arms

Cox model

For the CAV versus Topotecan comparison in the initial stage, the median and mean survival time figures for OAS in the CAV arm are respectively 5.84 and 4.48 months, and in the Topotecan arm 6.31 and 5.64 months. The difference in mean survival times is 1·16 months was associated with a standard deviation of 14.98 months. The hazard ratio for CAV versus Topotecan treatment is 0.89, with a confidence interval of 0.67 to 1.18. The associated P value is 0.418. Thus, by using simulated patient data, the Cox model demonstrated no difference in the survival distribution between CAV and Topotecan arms. See Fig. 5 for simulated survival times for the 2 study arms.

Fig. 5

Survival distribution for the CAV and Topotecan arms

However, for the II versus Topotecan comparison in the second stage, the median and mean survival times for OAS in the Topotecan arm are 6.30 and 6.42 months, and in the II arm 11.12 and 11.90 months, respectively, with a clear difference in mean survival times of 5.48 months associated with a standard deviation of 10.12 months. The hazard ratio for II versus Topotecan treatment is 0.44, with a confidence interval of 0.38 to 0.52. The associated P value is < 0.0001. This statistically difference in the survival of II versus Topotecan arms as shown by the Cox model is in accordance with the reinforcement learning findings that also demonstrated increased reward points with the II versus Topotecan arms. Refer to Fig. 6 for simulated survival times for individual patients, pointing at the superior survival distribution in the II simulated arm.

Fig. 6

Survival distribution for the Topotecan and Irinotecan Ifosfamide arms

Discussion

We demonstrate in this study that RL and patient simulation are suitable types of methodologies to test efficacy of various treatments in the field of oncology. The fact that we replicated the findings of a previously published phase 3 clinical trial in which CAV was compared with Topotecan, by means of patient simulation and Cox analysis, is encouraging to test new treatment options with the current standards by a similar methodology, at least to generate hypotheses for further clinical testing [4]. Additionally, for patients with extensive SCLC who progress after1st line chemotherapy, we show in this simulated study that II is likely to be associated with better OAS compared to the current standard Topotecan regimen, as reflected by the results of both RL and Cox analysis. Our findings are encouraging to further test II regimen versus Topotecan in recurrent or progressive SCLC in the context of a proper randomized clinical trial.

We think that the findings of this study provide insights into the evaluation of treatment strategies for small cell lung cancer (SCLC) using patient simulation and reinforcement learning techniques. The distribution of patient features in the simulated population closely resembled real-world clinical scenarios, with a male predominant representation of gender, a mean age of approximately 55, and a more frequent ECOG performance score of 0 or 1. These findings highlight the ability of patient simulation to generate representative virtual patient populations, allow interaction of patient features with the treatment efficacy, allowing for more accurate assessment of treatment outcomes.

The reinforcement learning analysis demonstrated significant differences in mean rewards between the Topotecan and II arms, with the II arm also outperforming the Topotecan arm in terms of OAS outcome as demonstrated by the Cox model. The difference in mean reward in these treatment arms is noteworthy, and the negative reward profile in the Topotecan arm suggests that, overall, survival was not significantly extended beyond the two-month threshold for most patients receiving Topotecan. In parallel, the mean survival difference between treatment arms is 5.48 months, albeit with a large standard deviation of 10.12 months, which raises the possibility of the magnitude of survival difference between the arms being overestimated by this analysis. Nevertheless, the difference in mean rewards, supported by the permutation test and other statistical tests, and findings from the Cox analysis that highlight a significant reduction in hazard ratio suggests that the II arm may offer a more favorable treatment option for SCLC patients. These findings are important for treatment decision-making and provide insights for optimizing and testing therapeutic approaches in clinical trials.

As stated above, the Cox proportional hazards regression model revealed a significant difference in survival outcomes between the II and Topotecan arms. The hazard ratio of 0.44 suggests a lower risk of death in the II arm compared to the Topotecan arm. These findings imply that II may be a superior treatment regimen in progressive SCLC. Obviously, our paper supports further research on the potential benefits of alternative treatment approaches for progressive SCLC, such as the use of immune checkpoint inhibitors, targeted therapies, antibody-drug conjugates, either alone, or in combination, as significant progress for relapsed SCLC is urgently required [25,26,27].

However, it is important to acknowledge some limitations of this study. Firstly, the simulation approach relies on assumptions and simplifications that may not capture the full complexity of real-world patient populations and treatment responses. Additionally, the simulation outcomes are based on the assumptions made for survival modifications and the use of randomization techniques. While efforts were made to align these assumptions with published data and known clinical factors, there is inherent uncertainty in the simulation results. In other words, due to the randomized confounders in the simulation, within the scope of the synthetic data—where treatment efficacy assumptions are embedded—the differences in survival can be attributed to the treatments themselves, not to any imbalance in confounders. However, we need to acknowledge that this is a reflection of the simulated model and not a direct claim about real-world efficacy. Furthermore, the study focused on a specific cohort of SCLC patients with recurrent disease and may not be directly generalizable to other patient populations. Secondly, we believe caution is necessary when assessing the comparability of overall survival (OAS) estimates, as they originate from different years and patient cohorts and are subsequently used as population parameters for simulating the treatment arms. Thirdly, we did not include the type of relapse or progression in the analysis (sensitive or refractory), and this could have caused bias in interpreting the results. Fourthly, in our current simulation, we did not explicitly account for the covariance structure among variables. The data were generated based on the individual distributions of each variable. While this approach reflects each variable’s characteristics, we recognize that future simulations could be improved by incorporating the covariance structure to more accurately reflect the complex relationships among variables in real-world cohorts. Fifthly, this analysis did not include treatment toxicity, which makes it harder to judge on the tolerability of the treatment protocols. Nevertheless, the toxicity profiles of these protocols from the published evidence suggest that both protocols are tolerable and manageable [4, 6].

However, as a last limitation of this study, we need to mention that survival times may not always follow a Weibull distribution, although previous work clearly shows that Weibull distribution is suitable to model oncology trials [18, 19]. Among alternative distributions are the exponential and log-normal distributions, which are also frequently used in survival modeling. We agree that further sensitivity analyses using multiple distribution assumptions could strengthen the robustness of our findings. In future work, we will explore alternative distributions and compare the results to ensure that the choice of distribution does not unduly influence our conclusions.

Medicine stands out as one area in which there is tremendous potential for AI along with equally substantial challenges. For the field of oncology in particular, AI is rapidly reshaping cancer research and personalized clinical care. Moving forward, further research is needed to refine and validate the simulation models and RL, which is a branch of AI, in the field of oncology as used in this study. Incorporating more comprehensive patient data, such as genetic markers, various patient and disease factors, and comorbidities, could enhance the accuracy and applicability of the simulations.

While reinforcement learning (RL) has the potential to capture various aspects of patient outcomes, in this study we focused specifically on survival as the primary endpoint. The RL approach allows us to evaluate treatment strategies by using reward differences, which in this context reflect survival outcomes across different time points. This dynamic evaluation contrasts with traditional methods that compare static survival measures like overall survival (OAS) or hazard ratios. By focusing on reward differences in survival, we offer a relatively novel method for assessing treatment efficacy over time, providing a more comprehensive understanding of survival benefits in relapsed SCLC in this paper, or potentially in different types of cancer in general.

This study contributes to the broader field of reinforcement learning (RL) in treatment research, particularly in its application to evaluating survival differences through the use of reward points. While RL is often employed for optimizing sequential decision-making in personalized medicine and dynamic treatment regimes, our focus here was on comparing treatment efficacy in terms of survival outcomes. By utilizing RL to model and differentiate survival benefits, this study highlights an alternative application of RL beyond its traditional use for real-time decision-making. This approach aligns with the growing interest in applying RL methodologies to assess treatment strategies in a more dynamic and nuanced way, offering valuable insights into treatment efficacy for different cancer types.

To sum up, this study demonstrates that a new protocol, II, has a potential to yield better survival than Topotecan in recurrent SCLC. After many years of being a standard treatment in the setting of recurrent SCLC, Topotecan is now challenged by a combination chemotherapy that is composed of Irinotecan and Ifosfamide. As progressive SCLC is rapidly fatal, new treatment options are eagerly awaited. Thus, our findings unveil a new treatment option in this context for SCLC.

In essence, our findings support the potential superiority of the II arm over the Topotecan arm in terms of survival outcomes in recurrent small cell lung cancer. Nonetheless, further research is needed to refine simulation models and validate findings in real-world settings. Secondly, this study demonstrates the utility of patient simulation and RL in evaluating treatment strategies for SCLC. Similar insights derived from using artificial intelligence techniques like RL, can guide treatment decision-making for other cancers as well, potentially leading to improved patient outcomes.

Conclusion

This study highlights the potential superiority of a new treatment protocol, II, over the standard Topotecan regimen in improving survival rates for recurrent small cell lung cancer (SCLC). The combination chemotherapy of Irinotecan and Ifosfamide presents a promising alternative to Topotecan, offering hope for better outcomes in a disease where effective treatments are urgently needed due to its aggressive nature. These findings underscore the importance of further research to refine simulation models and validate outcomes in real-world clinical settings. Moreover, the study demonstrates the value of patient simulation and reinforcement learning (RL) in evaluating treatment strategies for SCLC, with potential applications for enhancing patient care and outcomes in oncology.