Abstract
This study pioneers an innovative approach to improve the accuracy and dependability of emotion recognition (ER) systems by integrating electroencephalogram (EEG) with electrocardiogram (ECG) data. We propose a novel method of estimating effective connectivity (EC) to capture the dynamic interplay between the heart and brain during emotions of happiness, disgust, fear, and sadness. Leveraging three EC estimation techniques (Granger causality (GC), partial directed coherence (PDC) and directed transfer function (DTF)), we feed the resulting EC representations as inputs into convolutional neural networks (CNNs), namely ResNet-18 and MobileNetV2, known for their swift and superior performance. To evaluate this approach, we have used EEG and ECG data from the public MAHNOB-HCI database through 5-fold cross-validation criterion. Remarkably, our approach achieves an average accuracy of 97.34 ± 1.19 and 96.53 ± 3.54 for DTF images within the alpha frequency band using ResNet-18 and MobileNetV2, respectively. Comparative analyses in comparison to cutting-edge research unequivocally prove the efficacy of augmenting ECG with EEG, showcasing substantial improvements in ER performance across the spectrum of emotions studied.