Abstract
BACKGROUND: Millions of United States residents made use of public health websites during the coronavirus disease 2019 (COVID-19) pandemic to obtain information about vaccines and determine vaccine eligibility. OBJECTIVE: For public health websites to be effective, they must be usable. This study aimed to evaluate the usability of government websites for vaccine information, by examining whether these platforms helped users determine COVID-19 vaccine eligibility accurately and satisfactorily, as well as how health literacy (HL) plays a role in accurate eligibility judgments and satisfaction with the media. METHODS: A within-subject experiment was conducted (N = 39), where each participant used two websites and one embodied conversational agent system to determine their own eligibility for vaccination, as well as that of a fictitious persona as a standardized task. The website conditions in the study included the Centers for Disease Control (CDC) website, as well as vaccines.gov and mass.gov. The accuracy of participant-estimated eligibility and usability were further analyzed for association with HL. KEY RESULTS: Participants' estimate of vaccine eligibility was generally inaccurate for all website conditions, with an overall rate of 53.8% for correct responses. Participants with low HL had more incorrect responses and confusion, and HL was found to be a significant predictor of eligibility correctness when they were determining their own vaccine eligibility using the CDC website. Participants also reported having a significantly higher ability to find information and were more satisfied when interacting with the embodied conversational agent system, compared to the websites. CONCLUSIONS: Government websites-particularly the CDC website-were found to lack usability, especially for those with low HL. High error rates and low satisfaction underscore the need for simplification of public health site content and design and motivate the development of novel education methods for public health communication.