arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.02153cs.AIcs.CLcs.LG
收藏

通过去偏实现安全对齐语言模型的漏洞缓解

Vulnerability Mitigation for Safety-Aligned Language Models via Debiasing

LY Corporation;University of Tsukuba(筑波大学);RIKEN AIP(理化学研究所人工智能项目)

Thien Q. Tran, Akifumi Wachi, Rei Sato, Takumi Tanabe, Youhei Akimoto

AI总结:

本文针对安全对齐语言模型在特定类别中仍存在漏洞且可能偏向拒绝响应的问题,提出免学习的Token级安全去偏推理方法TSDI,在生成过程中纠正偏差,从而在保持安全性的同时提升模型有用性并改善Pareto前沿。

摘要:

安全对齐是现实世界人工智能应用中的一个重要研究课题。尽管人工智能中的安全性与可信赖性具有多面性,当前的安全对齐方法往往只关注一种综合性的安全概念。通过仔细评估现有安全对齐方法下的模型,我们发现,虽然它们总体上提升了安全性能,但未能确保特定类别中的安全性。我们的研究首先指出了在不牺牲模型有用性的前提下消除此类漏洞的困难。我们观察到,虽然较小的KL惩罚参数、增加训练迭代次数以及数据集清洗可以增强安全性,但它们并不一定能改善安全性与有用性之间的权衡。我们发现,安全对齐甚至可能引发不良效应,导致模型倾向于生成负面token,从而无论输入上下文如何都产生拒绝性响应。为了解决这一问题,我们引入了一种免学习的方法,即Token级安全去偏推理(TSDI),利用随机构造的提示在生成过程中估计并纠正这种偏差。我们的实验表明,该方法能够在保持安全性的同时提升模型的有用性,从而改善权衡的Pareto前沿。

英文摘要:

Safety alignment is an essential research topic for real-world AI applications. Despite the multifaceted nature of safety and trustworthiness in AI, current safety alignment methods often focus on a comprehensive notion of safety. By carefully assessing models from the existing safety-alignment methods, we found that, while they generally improved overall safety performance, they failed to ensure safety in specific categories. Our study first identified the difficulty of eliminating such vulnerabilities without sacrificing the model's helpfulness. We observed that, while smaller KL penalty parameters, increased training iterations, and dataset cleansing can enhance safety, they do not necessarily improve the trade-off between safety and helpfulness. We discovered that safety alignment could even induce undesired effects and result in a model that prefers generating negative tokens leading to rejective responses, regardless of the input context. To address this, we introduced a learning-free method, Token-level Safety-Debiased Inference (TSDI), to estimate and correct this bias during the generation process using randomly constructed prompts. Our experiments demonstrated that our method could enhance the model's helpfulness while maintaining safety, thus improving the trade-off Pareto-front.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑