arXiv 2504.096892025-04-29cs.AIcs.CLcs.CYcs.HCcs.LG

EmoAgent：评估与保障心理健康安全的人机交互

EmoAgent: Assessing and Safeguarding Human-AI Interaction for Mental Health Safety

Department of Electrical & Computer Engineering, Princeton University（普林斯顿大学电气与计算机工程系）；Department of Computer Science, Princeton University（普林斯顿大学计算机科学系）；Department of Computer Science & Engineering, University of Michigan（密歇根大学计算机科学与工程系）；Department of Data Science & Engineering, University of Michigan（密歇根大学数据科学与工程系）；Department of Philosophy, Columbia University（哥伦比亚大学哲学系）；AI Lab, Princeton University（普林斯顿大学人工智能实验室）；Chen Frontier Lab for Al and Mental Health, Tianqiao and Chrissy Chen Institute（天桥及克里斯西·陈研究所人工智能与心理健康前沿实验室）；Theta Health Inc.（Theta健康公司）

Jiahao Qiu, Yinghui He, Xinzhe Juan, Yimin Wang, Yuhan Liu, Zixin Yao, Yue Wu, Xun Jiang, Ling Yang, Mengdi Wang

备注：: 18 pages, 8 figures

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出多智能体框架EmoAgent，通过EmoEval评估LLM对脆弱用户的心理健康影响，并利用EmoGuard监控和纠正交互风险，显著降低了心理状态恶化率，保障人机交互安全。

摘要：

LLM驱动的AI角色的兴起引发了安全问题，特别是对于患有心理障碍的脆弱人类用户。为了应对这些风险，我们提出了EmoAgent，这是一个多智能体AI框架，旨在评估和缓解人机交互中的心理健康危害。EmoAgent包含两个组件：EmoEval模拟虚拟用户，包括扮演心理脆弱个体的用户，以评估与AI角色交互前后的心理健康变化。它使用临床验证的心理和精神评估工具（PHQ-9、PDI、PANSS）来评估LLM诱发的心理风险。EmoGuard作为中介，监控用户的心理状态，预测潜在危害，并提供纠正性反馈以缓解风险。在流行的基于角色的聊天机器人中进行的实验表明，情感投入的对话可能导致脆弱用户的心理恶化，在超过34.4%的模拟中出现了心理状态恶化。EmoGuard显著降低了这些恶化率，突显了其在确保更安全的AI与人类交互方面的作用。我们的代码可在以下网址获取：https://github.com/1akaman/EmoAgent

英文摘要：

The rise of LLM-driven AI characters raises safety concerns, particularly for vulnerable human users with psychological disorders. To address these risks, we propose EmoAgent, a multi-agent AI framework designed to evaluate and mitigate mental health hazards in human-AI interactions. EmoAgent comprises two components: EmoEval simulates virtual users, including those portraying mentally vulnerable individuals, to assess mental health changes before and after interactions with AI characters. It uses clinically proven psychological and psychiatric assessment tools (PHQ-9, PDI, PANSS) to evaluate mental risks induced by LLM. EmoGuard serves as an intermediary, monitoring users' mental status, predicting potential harm, and providing corrective feedback to mitigate risks. Experiments conducted in popular character-based chatbots show that emotionally engaging dialogues can lead to psychological deterioration in vulnerable users, with mental state deterioration in more than 34.4% of the simulations. EmoGuard significantly reduces these deterioration rates, underscoring its role in ensuring safer AI-human interactions. Our code is available at: https://github.com/1akaman/EmoAgent

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。