Abstract
Accurate tumor targeting is vital in prostate cancer (PCa) radiotherapy for precise dose delivery and minimizing healthy tissue exposure. Currently, no artificial intelligence (AI) tool exists to predict setup degradation risk considering rectal volume (RV) changes pre-treatment. Our study aimed to assess this risk and identify a critical RV cutoff. We retrospectively analyzed 498 cone-beam computed tomography (CBCT) scans from 38 PCa patients. Rectal organs at risk were contoured, and 3D couch shifts were calculated. A novel unsupervised 2-stage transformer-based encoder processed longitudinal RV and displacement data. K-means clustering grouped patients, visualized with T-distributed Stochastic Neighbor Embedding (t-SNE). Statistical analysis, including ROC AUC, identified an optimal planning CT RV cutoff. Three distinct patient clusters were observed in CBCT scans. A significant correlation (p < 0.001) was found between initial RVs and these clusters. The ROC AUC was 0.93, establishing an optimal cutoff of 81.62 cm³ for planning CT RV, effectively distinguishing patients prone to significant positional variability. Our study developed an AI-driven model predicting patient couch shifts in PCa radiotherapy, identifying a crucial RV threshold. These findings may advance positioning accuracy, enabling proactive adjustments and optimizing therapy. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1038/s41598-026-42276-7.