Abstract
Early diagnosis of skin cancer can significantly improve patient survival. Currently, skin lesion classification faces challenges such as lesion-background semantic entanglement, high intra-class variability, artifactual interference, and more, while existing classification models lack modeling of physicians' diagnostic paradigms. To this end, we propose DermViT, a medically driven deep learning architecture that addresses the above issues through a medically-inspired modular design. DermViT consists of three main modules: (1) Dermoscopic Context Pyramid (DCP), which mimics the multi-scale observation process of pathological diagnosis to adapt to the high intraclass variability of lesions such as melanoma, then extract stable and consistent data at different scales; (2) Dermoscopic Hierarchical Attention (DHA), which can reduce computational complexity while realizing intelligent focusing on lesion areas through a coarse screening-fine inspection mechanism; (3). Dermoscopic Feature Gate (DFG), which simulates the observation-verification operation of doctors through a convolutional gating mechanism and effectively suppresses semantic leakage of artifact regions. Our experimental results show that DermViT significantly outperforms existing methods in terms of classification accuracy (86.12%, a 7.8% improvement over ViT-Base) and number of parameters (40% less than ViT-Base) on the ISIC2018 and ISIC2019 datasets. Our visualization results further validate DermViT's ability to locate lesions under interference conditions. By introducing a modular design that mimics a physician's observation mode, DermViT achieves more logical feature extraction and decision-making processes for medical diagnosis, providing an efficient and reliable solution for dermoscopic image analysis.