arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.19574cs.HCcs.AIcs.CLcs.CYcs.SI
收藏

AI精神病:对话AI是否会放大妄想相关语言?

AI Psychosis: Does Conversational AI Amplify Delusion-Related Language?

University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)

Soorya Ram Shimgekar, Vipin Gunda, Jiwon Kim, Violeta J. Rodriguez, Hari Sundaram, Koustuv Saha

AI总结:

研究探讨对话AI对妄想相关语言的影响,通过模拟用户和三种模型家族分析妄想语言的变化,发现高妄想用户对话中妄想得分持续上升,而AI响应可有效降低此趋势。

摘要:

对话AI系统日益用于个人反思和情感披露,引发对脆弱用户影响的担忧。近期报告指出,长时间与AI互动可能强化妄想思维,即AI精神病。然而,此现象的实证证据有限。本文研究多轮对话中妄想相关语言的演变,构建了来自Reddit用户的SimUsers,并生成与三种模型家族(GPT、LLaMA和Qwen)的对话。开发了DelusionScore,量化对话轮次中的妄想相关语言强度。发现来自有先验妄想讨论用户(治疗组)的SimUsers表现出逐步上升的DelusionScore,而无此类讨论的用户(对照组)则保持稳定或下降。进一步发现,这种放大效应因主题而异,现实怀疑和强迫性推理显示最强增加。最后,根据当前DelusionScore调整AI响应可显著降低这些轨迹。这些发现提供了实证证据,表明对话AI互动可能在长期使用中放大妄想相关语言,并强调了状态感知安全机制的重要性,以缓解此类风险。

英文摘要:

Conversational AI systems are increasingly used for personal reflection and emotional disclosure, raising concerns about their effects on vulnerable users. Recent anecdotal reports suggest that prolonged interactions with AI may reinforce delusional thinking---a phenomenon sometimes described as AI Psychosis. However, empirical evidence on this phenomenon remains limited. In this work, we examine how delusion-related language evolves during multi-turn interactions with conversational AI. We construct simulated users (SimUsers) from Reddit users' longitudinal posting histories and generate extended conversations with three model families (GPT, LLaMA, and Qwen). We develop DelusionScore, a linguistic measure that quantifies the intensity of delusion-related language across conversational turns. We find that SimUsers derived from users with prior delusion-related discourse (Treatment) exhibit progressively increasing DelusionScore trajectories, whereas those derived from users without such discourse (Control) remain stable or decline. We further find that this amplification varies across themes, with reality skepticism and compulsive reasoning showing the strongest increases. Finally, conditioning AI responses on current DelusionScore substantially reduces these trajectories. These findings provide empirical evidence that conversational AI interactions can amplify delusion-related language over extended use and highlight the importance of state-aware safety mechanisms for mitigating such risks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑