Abstract
ChatGPT is not only fun to chat with, but it also searches information, answers questions, and gives advice. With consistent moral advice, it can improve the moral judgment and decisions of users. Unfortunately, ChatGPT's advice is not consistent. Nonetheless, it does influence users' moral judgment, we find in an experiment, even if they know they are advised by a chatting bot, and they underestimate how much they are influenced. Thus, ChatGPT corrupts rather than improves its users' moral judgment. While these findings call for better design of ChatGPT and similar bots, we also propose training to improve users' digital literacy as a remedy. Transparency, however, is not sufficient to enable the responsible use of AI.