Abstract
Advances in artificial intelligence (AI) have had a major impact on natural language processing (NLP), even more so with the emergence of large-scale language models like ChatGPT. This paper aims to provide a critical review of explainable AI (XAI) methodologies for AI chatbots, with a particular focus on ChatGPT. Its main objectives are to investigate the applied methods that improve the explainability of AI chatbots, identify the challenges and limitations within them, and explore future research directions. Such goals emphasize the need for transparency and interpretability of AI systems to build trust with users and allow for accountability. While integrating such interdisciplinary methods, such as hybrid methods combining knowledge graphs with ChatGPT, enhancing explainability, they also highlight industry needs for explainability and user-centred design. This will be followed by a discussion of the balance between explainability and performance, then the role of human judgement, and finally the future of verifiable AI. These are the avenues through which insights can be used to guide the development of transparent, reliable and efficient AI chatbots.