Abstract
BACKGROUND: Accurate nuclei segmentation in histopathological images is essential for quantitative pathology analysis but remains challenging due to nuclei clustering, morphological heterogeneity and weak or discontinuous boundaries. Although transformer-based models improve global context modeling, explicit enhancement of boundary-related features and effective suppression of false negatives (FNs) along nuclei edges are still limited. This study aimed to develop a boundary-enhanced nuclei segmentation framework by integrating high-frequency energy fusion (HFEF) with a sensitivity-aware loss. METHODS: We propose an HFEF framework that enhances nuclei boundary representation by integrating high-frequency components derived from the stationary wavelet transform (SWT) with Laplacian filtering. The resulting high-frequency energy map is fused as an additional input channel to enrich edge-related features in transformer-based segmentation models. In addition, a sensitivity-tuned loss (ST-Loss) is introduced to dynamically penalize low-confidence FNs, particularly in boundary regions. The proposed method was implemented in two edge-aware Swin Transformer-based segmentation models and evaluated with three publicly available histopathology and microscopy datasets. Performance was assessed using region-based metrics [Dice similarity coefficient (DSC) and intersection over union (IoU)] and boundary-aware metrics [Hausdorff distance (HDF) and boundary F1 score (BF1)]. Statistical significance was evaluated using paired one-tailed t-tests. RESULTS: Across three datasets and two model architectures, HFEF consistently improved nuclei segmentation performance compared with baseline configurations. DSC and IoU increased by 1.07% and 2.20%, respectively (P<0.05). Boundary delineation was substantially improved, with a 27% reduction in average HDF and a 0.89% increase in BF1. Incorporation of ST-Loss further increased recall by 2.48% (P=0.005), indicating effective suppression of boundary-related FNs. Visual inspection confirmed sharper nuclei boundaries and fewer missed detections, particularly in clustered and overlapping nuclei. CONCLUSIONS: The proposed HFEF framework, combined with sensitivity-aware loss optimization, enhances nuclei boundary representation and reduces false-negative errors, leading to improved segmentation accuracy and boundary precision. This approach is model-agnostic and has potential applicability to medical image segmentation tasks where accurate boundary delineation is critical. Source code is available at: https://github.com/deep-geo/HFEF.