Abstract
Reference scenarios based on mathematical models are used by public health experts to study infectious diseases. To gain insight into modeling assumptions, we analyzed the three major models that served as the basis for policy making in Israel during the COVID-19 pandemic and compared them to independently collected data. The number of confirmed patients, the number of patients in critical condition and the number of COVID-19 deaths predicted by the models were compared to actual data collected and published in the Israeli Ministry of Health's dashboard. Our analysis showed that the models succeeded in predicting the number of COVID-19 cases but failed to deliver an appropriate prediction of the number of critically ill and deceased persons. Inherent uncertainty and a multiplicity of assumptions that were not based on reliable information have led to significant variability among models, and between the models and real-world data. Although models improve policy leaders' ability to act rationally despite great uncertainty, there is an inherent difficulty in relying on mathematical models as reliable tools for predicting and formulating a strategy for dealing with the spread of an unknown disease.