arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.08240cs.CL
收藏

对齐之舞:联合训练代理以安全协作

The Alignment Waltz: Jointly Training Agents to Collaborate for Safety

Meta Superintelligence Labs(Meta超智能实验室);Johns Hopkins University(约翰·霍普金斯大学)

Jingyu Zhang, Haozhu Wang, Eric Michael Smith, Sid Wang, Amr Sharaf, Mahesh Pasupuleti, Benjamin Van Durme, Daniel Khashabi, Jason Weston, Hongyuan Zhan

AI总结:

本文提出WaltzRL框架,通过联合训练对话代理和反馈代理,解决LLM在安全与帮助性间的平衡问题,实验表明其有效降低不安全响应和过度拒绝率。

摘要:

利用LLM的力量需要在有益和无害之间进行微妙的平衡。这产生了一个根本性的矛盾:对抗攻击导致不安全内容的脆弱性,以及对良性但敏感提示的过度拒绝倾向。当前方法通常通过完全拒绝包含不安全内容的保障模型来导航这种平衡,但这种方法切断了音乐,可能加剧过度拒绝并无法提供针对拒绝查询的细致指导。为教模型更协调的舞蹈,我们提出了WaltzRL,一种新的多代理强化学习框架,将安全对齐作为协作、正和游戏来制定。WaltzRL联合训练对话代理和反馈代理,后者被激励提供有用的建议,以提高对话代理响应的安全性和有用性。WaltzRL的核心是一个动态改进奖励(DIR),其根据对话代理如何整合反馈而演变。在推理时,对话代理的不安全或过度拒绝响应被改进而不是丢弃。反馈代理与对话代理一起部署,并且只有在需要时才适应性地参与,从而在安全查询上保持帮助性和低延迟。我们的实验在五个多样化的数据集上进行,表明WaltzRL显著降低了不安全响应(例如,在WildJailbreak上从39.0%降至4.6%)和过度拒绝(在OR-Bench上从45.3%降至9.9%)与各种基线相比。通过使对话和反馈代理能够共同进化和适应性地应用反馈,WaltzRL在不降低通用能力的情况下增强了LLM的安全性,从而在有益性和无害性之间推进帕累托前沿。

英文摘要:

Harnessing the power of LLMs requires a delicate dance between being helpful and harmless. This creates a fundamental tension between two competing challenges: vulnerability to adversarial attacks that elicit unsafe content, and a tendency for overrefusal on benign but sensitive prompts. Current approaches often navigate this dance with safeguard models that completely reject any content that contains unsafe portions. This approach cuts the music entirely-it may exacerbate overrefusals and fails to provide nuanced guidance for queries it refuses. To teach models a more coordinated choreography, we propose WaltzRL, a novel multi-agent reinforcement learning framework that formulates safety alignment as a collaborative, positive-sum game. WaltzRL jointly trains a conversation agent and a feedback agent, where the latter is incentivized to provide useful suggestions that improve the safety and helpfulness of the conversation agent's responses. At the core of WaltzRL is a Dynamic Improvement Reward (DIR) that evolves over time based on how well the conversation agent incorporates the feedback. At inference time, unsafe or overrefusing responses from the conversation agent are improved rather than discarded. The feedback agent is deployed together with the conversation agent and only engages adaptively when needed, preserving helpfulness and low latency on safe queries. Our experiments, conducted across five diverse datasets, demonstrate that WaltzRL significantly reduces both unsafe responses (e.g., from 39.0% to 4.6% on WildJailbreak) and overrefusals (from 45.3% to 9.9% on OR-Bench) compared to various baselines. By enabling the conversation and feedback agents to co-evolve and adaptively apply feedback, WaltzRL enhances LLM safety without degrading general capabilities, thereby advancing the Pareto front between helpfulness and harmlessness.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑