Abstract
BACKGROUND: ChatGPT's potential as a diet information tool is emerging. However, little is known about the extent to which the information provided by ChatGPT aligns with that provided by dietitians. OBJECTIVE: This study aimed to assess ChatGPT's capacity to provide responses to diet-related questions, compared to responses by dietitians. METHODS: A total of 928 diet-related questions and corresponding responses from dietitians were collected from Naver Knowledge-iN, a Korean online Q&A platform, between January 18, 2023, and January 17, 2024. ChatGPT-4o was used to generate responses to the same questions. Five text similarity indices-Dice Coefficient, Jaccard Index, Overlap Coefficient, Cosine Similarity, and Term Frequency-Inverse Document Frequency-were used to assess the similarity between ChatGPT's and dietitians' responses. Questions with the top 5% response similarity were reviewed to identify characteristics of the questions for which ChatGPT generated responses similar to those of dietitians. Responses with the bottom 5% similarity were reviewed to identify reasons for the low similarity. RESULTS: The average similarity coefficient between ChatGPT and dietitian responses was 0.42. Questions with high response similarity tended to include detailed information, such as specific food items or portions (76.1%), the questioner's context (69.6%), or personal characteristics (17.4%). Low response similarity was mainly due to ChatGPT providing significantly longer responses than dietitians. CONCLUSIONS: ChatGPT demonstrated content similarity to dietitian responses, but they were not identical. The development of prompt engineering techniques to enhance ChatGPT's ability to provide more expert-like and personalized information could benefit users seeking dietary information.