Abstract
Optical Coherence Tomography (OCT) plays a crucial role in diagnosing ocular diseases, yet conventional CNN-based models face limitations such as high computational overhead, noise sensitivity, and data imbalance. This paper introduces HDL-ACO, a novel Hybrid Deep Learning (HDL) framework that integrates Convolutional Neural Networks with Ant Colony Optimization (ACO) to enhance classification accuracy and computational efficiency. The proposed methodology involves pre-processing the OCT dataset using discrete wavelet transform and ACO-optimized augmentation, followed by multiscale patch embedding to generate image patches of varying sizes. The hybrid deep learning model leverages ACO-based hyperparameter optimization to enhance feature selection and training efficiency. Furthermore, a Transformer-based feature extraction module integrates content-aware embeddings, multi-head self-attention, and feedforward neural networks to improve classification performance. Experimental results demonstrate that HDL-ACO outperforms state-of-the-art models, including ResNet-50, VGG-16, and XGBoost, achieving 95% training accuracy and 93% validation accuracy. The proposed framework offers a scalable, resource-efficient solution for real-time clinical OCT image classification.