Abstract
Breast cancer remains a global health challenge, ranking as a leading cause of mortality among women worldwide, emphasizing the need for early detection to improve treatment outcomes. Architectural distortion (AD), characterized by subtle disruptions in breast tissue patterns, is one of the earliest and most challenging signs of breast cancer to detect, often manifesting up to 2 years before other signs and providing a critical window for intervention. To address the limitations of traditional AD segmentation methods, this study introduces an advanced deep learning approach to automate and improve AD segmentation and classification on mammograms, significantly reducing radiologists' workload and enhancing diagnostic accuracy. The process began with implementing U-Net++ as a semantic segmentation model for pixel-level classification. Next, a standalone Mask R-CNN was applied, which incorporated instance segmentation for more precise detection of AD regions. Finally, the study introduced a two-phase pipeline that combines Mask R-CNN for segmentation with a ResNet-18 classification model to refine predictions and reduce false positives. Key enhancements, such as smooth L1 for bounding box regression loss and binary cross entropy with Dice loss for mask prediction, significantly improved segmentation metrics. The integrated approach achieved remarkable results with a segmentation accuracy of 0.852, a classification accuracy of 0.915, and a mean average precision (mAP) of 0.894. Furthermore, the sensitivity of our integrated approach was 92.4%. This enhances breast cancer screening and diagnosis, with the results highlighting its potential to improve patient outcomes through timely diagnosis and effective treatment planning.