发表机构
Umeå University; Univ. Grenoble Alpes, CNRS, Grenoble INP, LJK（于默奥大学; 格勒诺布尔阿尔卑斯大学、法国国家科学研究中心、格勒诺布尔国立理工学院、让·昆斯特勒实验室）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对开放权重模型安全对齐中的过度拒绝和质量下降问题，提出梯度级偏好优化算法Suan，在保持响应效用的同时实现卓越安全对齐。
AI 中文摘要
将稳健的安全护栏集成到大语言模型（LLMs）中，对于提供有用且无害的响应至关重要。尽管专有系统表现出可靠的安全控制，但其底层方法和权衡在很大程度上仍未公开。在开放权重模型中实现同等的安全性仍然是一个持续的挑战，因为经过后训练的变体经常遭受过度拒绝和通用质量下降的问题。为了克服这些缺点，我们引入了Suan，一种新颖的偏好优化算法。与现有方法不同，我们直接在梯度层面制定优化目标，绕过了标准的变分推导。因此，我们获得了更具可解释性和稳健性的训练动态。在多种竞争性基线和基准上的广泛评估表明，Suan在完全保留响应效用的同时，实现了卓越的安全对齐。
英文摘要
Integrating robust safety guardrails into Large Language Models (LLMs) is essential for delivering helpful yet harmless responses. While proprietary systems exhibit reliable safety controls, their underlying methodologies and trade-offs remain largely undisclosed. Achieving comparable security in open-weight models remains a persistent challenge, as post-trained variants frequently suffer from over-refusal and degraded general quality. To overcome these drawbacks, we introduce Suan, a novel preference optimization algorithm. Unlike existing methods, we formulate the optimization objective directly at the gradient level, bypassing the standard variational derivation. As a result, we obtain more interpretable and robust training dynamics. Extensive evaluations across a diverse suite of competitive baselines and benchmarks demonstrate that Suan achieves superior safety alignment while fully preserving response utility.
