Abstract
We examine how convolutional neural networks (CNNs) for cardiac rhythm device detection can exhibit failures in performance under suboptimal deployment scenarios and examine how medically adversarial image presentation can further impair neural network performance. We validated the publicly available Pacemaker-ID web server and mobile app on 43 local hospital emergency department (ED) cases of patients presenting with a cardiac rhythm device on anterior-posterior (AP) chest radiograph and assessed performance using Cohen's kappa coefficient for inter-rater reliability. To illustrate adversarial performance concerns, we then produced example CNN models using the 65,379 patient MIMIC-CXR chest radiograph retrospective database and evaluated performance with area under the receiver operating characteristic (AUROC). In retrospective review of 43 patients with cardiac rhythm devices on AP chest radiographs during our study period (January 1, 2020 to March 1, 2020), 74.4% (32/43) had device manufacturer information readily available within the electronic medical record. A total of 25.6% of patients (11/43) did not have this information documented in the patient chart and could ostensibly benefit from CNN-based identification of device manufacturer. For patients with known device manufacturer, the Pacemaker-ID prediction was accurate in 87.5% of cases (28/32). Mobile app accuracy varied from 62.5 to 93.75% depending on image capture settings and presentation. Cohen's kappa coefficient varied from 0.448 to 0.897 depending on mobile image capture conditions. For our additional analysis of medically adversarial performance failures with a DenseNet121 trained on MIMIC-CXR images, we showed that an AUROC of 0.9807 ± 0.0051 could be achieved on an example testing dataset while masking a 30% false positive rate in identification of cardiac rhythm devices versus clinically distinct entities such as vagal nerve stimulators. Despite the promise of CNN approaches for cardiac rhythm device analysis on chest radiographs, further study is warranted to assess potential for errors driven by user misuse when deploying these models to mobile devices as well as for cases when performance can be impaired by the presence of other support apparatuses.