Abstract
INTRODUCTION: Manual identification of case narratives with specific relevant information can be challenging when working with large numbers of adverse event reports (case series). The process can be supported with a search engine, but building search queries often remains a manual task. Suggesting terms to add to the search query could support assessors in the identification of case narratives within a case series. OBJECTIVE: The aim of this study is to explore the feasibility of identifying case narratives containing specific characteristics with a narrative search engine supported by artificial intelligence (AI) query suggestions. METHODS: The narrative search engine uses Best Match 25 (BM25) and suggests additional query terms from two word embedding models providing English and biomedical words to a human in the loop. We calculated the percentage of relevant narratives retrieved by the system (recall) and the percentage of retrieved narratives relevant to the search (precision) on an evaluation dataset including narratives from VigiBase, the World Health Organization global database of adverse event reports for medicines and vaccines. Exact-match search and BM25 search with the Relevance Model (RM3), an alternative way to expand queries, were used as comparators. RESULTS: The gold standard included 55/750 narratives labelled as relevant. Our narrative search engine retrieved on average 56.4% of the relevant narratives (recall), which is higher when compared with exact-match search (21.8%), without a significant drop in precision (54.5% to 43.1%). The recall is also higher as compared with RM3 (34.4%). CONCLUSIONS: Our study demonstrates that a narrative search engine supported by AI query suggestions can be a viable alternative to an exact-match search and BM25 search with RM3, since it can facilitate the retrieval of additional relevant narratives during signal assessments.