arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2512.24263cs.AI
收藏

通过风险感知的逐步对齐实现约束语言模型策略优化

Constrained Language Model Policy Optimization via Risk-aware Stepwise Alignment

Key Laboratory of Computational Intelligence and Chinese Information Processing of Ministry of Education, School of Computer and Information Technology, Shanxi University(教育部计算智能与中文信息处理重点实验室,计算机与信息技术学院,山西大学);University College London(伦敦大学学院);Institute for AI, Peking University(北京大学人工智能研究院)

Lijun Zhang, Lin Li, Wei Wei, Yajie Qi, Huizhong Song, Jun Wang, Yaodong Yang, Jiye Liang

AI总结:

本文提出风险感知的逐步对齐方法,通过嵌套风险度量提升语言模型策略优化的安全性与鲁棒性,有效抑制高影响有害行为。

摘要:

在微调预训练语言模型(LMs)以表现出期望行为时,保持对风险的控制对于确保安全性和可信度至关重要。现有的大多数安全对齐方法,如Safe RLHF和SACPO,通常基于风险中性范式,无法解决偏离参考策略带来的风险,并且对罕见但可能造成灾难性有害行为的鲁棒性有限。为了解决这一限制,我们提出了风险感知的逐步对齐(RSA),一种新的对齐方法,通过利用一类嵌套风险度量,将风险意识显式地纳入策略优化过程中。具体而言,RSA将安全对齐制定为一个token级风险感知的约束策略优化问题,并通过逐步对齐程序解决,该程序通过嵌套风险度量产生token级策略更新。这种设计提供了两个关键优势:(1)它减轻了模型过度偏离参考策略引起的 risks,(2)它明确抑制了低概率但高影响的有害行为。此外,我们还提供了在温和假设下的策略最优性理论分析。实验结果表明,我们的方法在保持高帮助性的同时,确保了强安全性,并显著抑制了尾部风险,即低概率但高影响的不安全响应。

英文摘要:

When fine-tuning pre-trained Language Models (LMs) to exhibit desired behaviors, maintaining control over risk is critical for ensuring both safety and trustworthiness. Most existing safety alignment methods, such as Safe RLHF and SACPO, typically operate under a risk-neutral paradigm that is insufficient to address the risks arising from deviations from the reference policy and offers limited robustness against rare but potentially catastrophic harmful behaviors. To address this limitation, we propose Risk-aware Stepwise Alignment (RSA), a novel alignment method that explicitly incorporates risk awareness into the policy optimization process by leveraging a class of nested risk measures. Specifically, RSA formulates safety alignment as a token-level risk-aware constrained policy optimization problem and solves it through a stepwise alignment procedure that yields token-level policy updates derived from the nested risk measures. This design offers two key benefits: (1) it mitigates risks induced by excessive model shift away from a reference policy, and (2) it explicitly suppresses low-probability yet high-impact harmful behaviors. Moreover, we provide theoretical analysis on policy optimality under mild assumptions. Experimental results demonstrate that our method achieves high levels of helpfulness while ensuring strong safety and significantly suppresses tail risks, namely low-probability yet high-impact unsafe responses.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑