Abstract
Assessing lymph node metastasis (LNM) involvement in patients with rectal cancer (RC) is fundamental in disease management. In this study, we used artificial intelligence (AI) technology to develop a segmentation model that automatically segments the tumor core area and mesangial tissue from magnetic resonance T2-weighted imaging (T2WI) and apparent diffusion coefficient (ADC) images collected from 122 RC patients to improve the accuracy of LNM prediction, after which omics machine modeling was performed on the segmented ROI. An automatic segmentation model was developed using nn-UNet. This pipeline integrates deep learning (DL), specifically 3D U-Net, for semantic segmentation and image processing techniques such as resampling, normalization, connected component analysis, image registration, and radiomics features coupled with machine learning. The results showed that the DL segmentation method could effectively segment the tumor and mesangial areas from MR sequences (the median dice coefficient: 0.90 ± 0.08; mesorectum segmentation: 0.85 ± 0.36), and the radiological characteristics of rectal and mesangial tissues in T2WI and ADC images could help distinguish RC treatments. The nn-UNet model demonstrated promising preliminary results, achieving the highest area under the curve (AUC) values in various scenarios. In the evaluation encompassing both tumor lesions and mesorectum involvement, the model exhibited an AUC of 0.743, highlighting its strong discriminatory ability to predict a combined outcome involving both elements. Specifically targeting tumor lesions, the model achieved an AUC of 0.731, emphasizing its effectiveness in distinguishing between positive and negative cases of tumor lesions. In assessing the prediction of mesorectum involvement, the model displayed moderate predictive utility with an AUC of 0.753. The nn-UNet model demonstrated impressive performance across all evaluated scenarios, including combined tumor lesions and mesorectum involvement, tumor lesions alone, and mesorectum involvement alone. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s12880-025-01878-9.