Abstract
BACKGROUND: There is a shortage of services available to address the growing demand for mental health support in Australia and worldwide. Digital interventions, including conversational agents, can overcome barriers to accessing mental health support. The recent advances in large language models have led to an improvement in the perceived human-like naturalness of chatbot conversations, but there is little research on the experience of chatbots to support mental health. Manage your life online (MYLO) is a rule-based chatbot that was co-designed with young people and uses questions to help users explore their problems. In a case series conducted before the release of ChatGPT (OpenAI), users rated a new smartphone interface for MYLO as acceptable, and there was a large effect size for reduction in problem-related distress. OBJECTIVE: This study aimed to evaluate an improved version of MYLO and compare the user experience of MYLO in this case series to the previous version that was completed in November 2022. METHODS: We replicated and extended the previous 2-week case-series, conducted from September to November 2022, by testing 4-week usage of MYLO with a larger sample between October and December 2023. We recruited 24 young people living in Western Australia who self-described as having a lived experience of anxiety or depression. Participants had access to and used MYLO over a 4-week period while completing online weekly surveys that included a range of health and psychological questionnaires. After the 4-week testing phase, participants were invited to provide feedback on their experience of using MYLO through an interview or focus group discussion. RESULTS: In total, 13 of the 24 participants were retained throughout the study and took part in interviews. On average, participants had around 4 conversations with MYLO. They experienced both benefits and limitations of these conversations. They spoke about their recent experiences with ChatGPT (released in November 2022 after the previous case-series concluded) and other generative artificial intelligence (AI) tools, stating that they had expected MYLO to possess similar functionality, which it did not. Nonetheless, we found moderate to large effect sizes for improvements in problem-related distress (Cohen d=-1.07), anxiety (Cohen d=-0.41), and psychiatric impairment (Cohen d=-0.60) and some evidence of reliable improvement in clinical outcomes. CONCLUSIONS: These findings have implications for mental health chatbots in the age of ChatGPT and highlight a need for researchers to engage with new technologies to improve user experience, while maintaining the necessary safety and ethical standards that can be a significant challenge for generative AI.