Abstract
Artificial Intelligence (AI) systems are increasingly being deployed across various high-risk applications, especially in healthcare. Despite significant attention to evaluating these systems, post-deployment incidents are not uncommon, and effective mitigation strategies remain challenging. Drug safety has a well-established history of assessing, monitoring, understanding, and preventing adverse effects in real-world usage, known as pharmacovigilance. Drawing inspiration from pharmacovigilance methods, we discuss concepts that can be adapted for monitoring AI systems in healthcare. This discussion aims to improve responses to adverse effects and potential incidents and risks associated with AI deployment in healthcare but also beyond.