Abstract
Driver fatigue is a significant factor contributing to road accidents, highlighting the need for precise and reliable detection systems. This study introduces a comprehensive approach using multimodal neural networks, leveraging the DROZY dataset, which includes physiological and facial data collected under sleep deprivation conditions. Two advanced neural network models were developed and evaluated: a multimodal feature combination model and a multimodal feature coupled model. The multimodal feature coupled model, which is the highlight of this study, uniquely integrates various data types, such as electroencephalograms (EEG), electrocardiograms (ECG), and facial images, through a sophisticated coupling mechanism. In this model, the features from different modalities are not merely combined; instead, they serve as mutual weights, influencing each other's contribution to the final prediction. This dynamic interaction between features enhances the model's ability to capture complex patterns associated with driver fatigue, leading to superior performance. The model achieved an outstanding accuracy of 98.41%, precision of 98.38%, recall of 98.39%, and an F1-score of 98.38%, demonstrating its exceptional capability in accurately detecting driver fatigue. In comparison, the multimodal feature combination model also performed well, with an accuracy of 94.87%, precision of 95.18%, recall of 95.04%, and an F1-score of 95.00%. To ensure robust decision-making in real-world applications, a majority voting strategy was employed in the decision phase, aggregating predictions from multiple classifiers to provide consistent and reliable fatigue warnings. These results highlight the advantages of the multimodal feature-coupled model in addressing the challenges of driver fatigue detection, making it a valuable tool for enhancing road safety through advanced, efficient monitoring systems in vehicles.