Abstract
The automated diagnosis of low-resolution and difficult-to-recognize breast ultrasound images through multi-modal fusion holds significant clinical value. However, prevailing fusion methods predominantly rely on image modalities, neglecting the textual pathology information, and only benign and malignant diagnosis of breast tumors is not satisfying for clinical applications. Consequently, this paper proposes a novel multi-modal fusion interactive diagnostic framework, termed the MIC framework, to achieve the multi-label classification of breast cancer, namely benign-malignant classification and breast imaging reporting and data system (BI-RADS) 3, 4a, 4b, 4c, and 5 gradings. The framework fusions brightness-mode ultrasound images, contrast-enhanced ultrasound images, and pathological information. Firstly, the two image modalities through the multi-modal similarity module capture inter-modal high similarity features. Secondly, the interactive feature enhancement module extracts abundant global-local and multi-scale complementary information from the images. Thirdly, pathological information is fused through the cross-modal interaction module to achieve the combination of image data and pathology knowledge. Finally, parallel classifier and joint loss function are used to realize and optimize the multi-label classification. Experimental results show that the proposed framework achieves accuracy, precision, recall, and F1 of 98.45%, 98.25%, 98.06%, and 98.43%, respectively, demonstrating it can recognize breast cancers effectively.