arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.00950cs.AI
收藏

MindGuard: 多轮心理健康支持中的防护分类器

MindGuard: Guardrail Classifiers for Multi-Turn Mental Health Support

Sword Health;Instituto de Telecomunicações(电信研究所);Instituto Superior Técnico(高等技术学院)

António Farinhas, Nuno M. Guerreiro, José Pombal, Pedro Henrique Martins, Laura Melton, Alex Conway, Cara Dochat, Maya D'Eon, Ricardo Rei

AI总结:

MindGuard通过临床验证的危险分类法,提升多轮心理健康对话中的安全性和准确性。

摘要:

大型语言模型越来越多地用于心理健康支持,但仅凭对话连贯性并不能保证临床适当性。现有通用安全措施往往无法区分治疗性披露和真实临床危机,导致安全故障。为解决这一差距,我们引入了一个基于临床的危险分类法,该分类法与博士级心理学家合作开发,能够识别可操作的伤害(例如自伤和伤害他人),同时保留安全、非危机治疗性内容的空间。我们发布了MindGuard-testset,这是一个由临床专家在对话层面注释的现实世界多轮对话数据集。通过使用由受控双代理设置生成的合成对话,我们训练了MindGuard,即一组轻量级的安全分类器(参数量为4B和8B)。我们的分类器在高召回率操作点上减少了误报,当与临床语言模型结合使用时,在对抗性多轮交互中能实现更低的攻击成功率和有害参与率,相比通用安全措施。我们发布了所有模型和人类评估数据。

英文摘要:

Large language models are increasingly used for mental health support, yet their conversational coherence alone does not ensure clinical appropriateness. Existing general-purpose safeguards often fail to distinguish therapeutic disclosures from genuine clinical crises, and progress on developing better ones is bottlenecked by a lack of clinically grounded evaluation resources. To address this gap, we introduce a risk taxonomy, developed in collaboration with PhD-level clinical psychologists, that identifies actionable harm (self-harm and harm to others) while preserving space for safe, non-crisis therapeutic content. We release MindGuard-testset, a dataset of multi-turn conversations annotated at the turn level by clinical experts, which is, to our knowledge, the first public benchmark with turn-level clinical risk labels for multi-turn mental health support. We also release an automated red-teaming (ART) framework designed to measure how safety classifiers affect downstream model behavior in adversarial multi-turn interactions. Using synthetic dialogues generated via a controlled two-agent setup, we train MindGuard, a family of lightweight safety classifiers (with 4B and 8B parameters). Our classifiers reduce false positives at high-recall operating points and, when paired with clinician language models, help achieve lower attack success and harmful engagement rates compared to general-purpose safeguards. We release all models, training and human evaluation data, and the ART framework.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑