Abstract
BACKGROUND AND PURPOSE: Deep-learning auto-segmentation (DLAS) performance in radiotherapy may change over time due to data shift/drift or practice changes, yet guidance for quality assurance is lacking. This study developed a practical framework for prospective performance monitoring using retrospective data. METHODS: A total of 464 prostate cases over 20 months were retrospectively collected. Two commercial DLAS models were clinically used: model A (2D U-Net, January 2022-January 2023) and model B (3D U-Net, February-August 2023). The agreement between DLAS and clinical contours was assessed using Dice Similarity Coefficient (DSC), 95th percentile Hausdorff Distance (HD95), and Surface DSC with a 2 mm tolerance (SDSC). Statistical process control charts were created to monitor performance drift and model switching. The first 150 cases were used to define organ-specific control limits with two and three standard deviations of monthly mean values, σx¯ . RESULTS: 2 σx¯ and 3 σx¯ -based control limits were established for the monthly average charts, ranging from DSC 0.82-0.97, HD95 1.4-10.5 mm, and SDSC 0.45-0.91 across organs. Model A showed stable performance, with 9-13 months per organ remaining within the 3 σx¯ thresholds. In contrast, model B demonstrated a marked performance shift (p < 0.001), with all five organs exceeding both thresholds across all 7 months. The 2 σx¯ thresholds were more sensitive in detecting mild deviations for model A, while both limits effectively identified the substantial drift of model B. CONCLUSION: The monitoring system effectively detected out-of-distribution outliers and clinical practice changes, providing a reliable framework for early detection of monthly performance degradation.