Abstract
Brain tumor segmentation is a crucial yet complex task in medical image analysis, playing a vital role in accurate diagnosis and treatment planning. However, it faces several inherent challenges: (1) class imbalance, where tumors occupy a minimal portion of MRI scans, leading models to favor healthy tissue; (2) low sensitivity to small or irregular tumors, resulting in high false negatives; and (3) limitations of standard loss functions, which often fail to properly emphasize tumor boundaries.TraditionalHausdorff Loss, despite its effectiveness in boundary alignment, suffers from non-differentiability and high sensitivity to outliers, leading to unstable optimization and poor boundary predictions. To address these challenges, we propose a novel Sub-Differentiable Hausdorff Loss (SDHL), which introduces a smooth, differentiable formulation enabling stable gradient-based learning and robust boundary alignment. Furthermore, to balance fine boundary precision with accurate tumor region segmentation, we introduce a novel SDHL combined with BCE loss. This combination leverages SDHL's boundary sensitivity and BCE's region accuracy, resulting in sharper tumor boundaries and more complete segmentation outcomes. This study presents a deep learning-based brain tumor segmentation pipeline utilizing various UNet-derived architectures including UNet, UNet+, VNet, UNet++, and Attention UNet, all trained using the proposed loss functions. The models evaluated using performance metrics such as Accuracy, Precision, Recall, Dice Score, and Intersection over Union (IoU). Among the tested configurations, Semantic segmentationmetricsofAttention UNet with SDHL + BCE outperformed all others, achieving 99.71% accuracy, 91.06% precision, 91.85% recall, 90.16% Dice score, and 80.78% IoU. The integration of the proposed loss functions significantly improves segmentation quality, addressing key challenges and offering a robust solution for brain tumor segmentation in medical imaging.Additionally, the Attention UNet model demonstrated strong consistency across evaluation metrics, achieving 97.92% accuracy, 98.76% precision, 98.05% recall, 98.57% F1-score, and a Matthews correlation coefficient of 93.41%, indicating robust and reliable performance.