Abstract
BACKGROUND: Accurate assessment of axillary lymph node status is essential for the management of breast cancer. Recent advancements in deep learning (DL) have shown promising results in medical image analysis. This study aims to develop a multimodal DL model that integrates preoperative ultrasound images and hematoxylin and eosin (H&E)-stained core needle biopsy pathology images of primary breast cancer to predict axillary lymph node metastasis (ALNM). MATERIALS AND METHODS: This study included 211 patients with histologically confirmed breast cancer, conducted between February 2023 and March 2024. For each patient, one ultrasound image and one histopathological image of the primary breast cancer lesion were collected. Various DL architectures were applied to extract tumor features from the ultrasound and histopathology images, respectively. Multiple fusion strategies, combining features from both ultrasound and pathology images, were developed to enhance the comprehensiveness and accuracy of predictions. The performance of the single-modality models, multi-modality models, and different fusion strategies were compared. Evaluation metrics included precision, accuracy, recall, F1-score, and area under the curve (AUC). RESULTS: PLNeT and ULNet were identified as the most effective feature extractors for histopathological and ultrasound image analysis, respectively. Overall, the multilayer fusion model outperformed single-modality models in predicting ALNM, achieving an accuracy of 0.7353, precision of 0.7344, recall of 0.7576, F1-score of 0.7463, and AUC of 0.7019. CONCLUSION: Our study provides a multilayer fusion strategy using ultrasound and pathology images of the primary tumor to predict ALNM in breast cancer patients. Although achieving suboptimal performance, this model has the potential to determine appropriate axillary treatment options for patients with breast cancer.