Abstract
Objective: Radiotherapy is a primary method for cancer treatment, wherein radiation doses are divided into multiple sessions or fractions to effectively target tumors and minimize damage to surrounding tissues. Methods: In this study, we leverage reinforcement learning (RL) to enhance treatment planning with the aim of improving the adaptability and robustness of RL agents given the inherent inaccuracies in tumor growth models. A 2D simulation model of tumor growth is employed, where tabular RL techniques are used to determine the optimal treatment strategies. We emphasize the significance of tissue damage predictions and incorporate the Lyman NTCP model to assess treatment outcomes, analyzing complications across three simulated body sites: the rectum, head and neck and lung. Results: For all the tumor sites, the RL approach significantly reduces healthy tissue damage by 10.7%, 49.1% and 37.5%, respectively, for rectal, head and neck and lung cancers compared with the baseline treatment. Conclusions: The RL-based approach in radiotherapy not only achieves tumor eradication but also significantly reduces healthy tissue damage compared with traditional treatment methods. This study demonstrates the potential of reinforcement learning to optimize treatment planning in radiotherapy, offering a promising path towards more personalized and effective cancer treatments.