arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.02136cs.AI
收藏

通过基于分布的细化缓解安全税在大推理模型中的影响

Mitigating Safety Tax via Distribution-Grounded Refinement in Large Reasoning Models

School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University, China(中山大学深圳校区计算机科学与技术学院);Hong Kong University of Science(香港科技大学)

Yingsha Xie, Tiansheng Huang, Enneng Yang, Rui Min, Wenjie Lu, Xiaochun Cao, Naiqiang Tan, Li Shen

AI总结:

本文提出DGR方法,通过基于分布的细化缓解大推理模型中的安全税,提升推理准确性和安全性。

摘要:

安全对齐会带来安全税,这会干扰大推理模型(LRM)的通用推理能力。现有用于大推理模型的安全对齐数据集通常是通过从外部大推理模型或人工标注者中蒸馏安全推理轨迹和答案构建的。然而,这些推理轨迹和答案与需要对齐的目标大推理模型存在分布差距,我们推测这种分布差距是导致目标大推理模型推理能力显著下降的罪魁祸首。基于这一假设,我们提出了一种安全对齐数据集构建方法,称为DGR。DGR将现有的分布不一致的安全推理数据集转换并细化,以与目标LLM的内部分布对齐。实验结果表明,i)DGR在所有基线中有效缓解了安全税,同时保持了安全性表现,即在DirectRefusal上实现了+30.2%的提升,在平均推理准确性上相比Vanilla SFT提升了+21.2%;ii)推理能力下降的程度与分布偏移的程度相关,这表明弥合这一差距是保持能力的关键。此外,我们发现大推理模型中的安全对齐可能主要作为一种机制来激活潜在知识,因为仅需10个样本即可激活有效的拒绝行为。这些发现不仅强调了分布一致性的重要性,还为推理模型中安全机制的激活机制提供了见解。

英文摘要:

Safety alignment incurs safety tax that perturbs a large reasoning model's (LRM) general reasoning ability. Existing datasets used for safety alignment for an LRM are usually constructed by distilling safety reasoning traces and answers from an external LRM or human labeler. However, such reasoning traces and answers exhibit a distributional gap with the target LRM that needs alignment, and we conjecture such distributional gap is the culprit leading to significant degradation of reasoning ability of the target LRM. Driven by this hypothesis, we propose a safety alignment dataset construction method, dubbed DGR. DGR transforms and refines an existing out-of-distributional safety reasoning dataset to be aligned with the target's LLM inner distribution. Experimental results demonstrate that i) DGR effectively mitigates the safety tax while maintaining safety performance across all baselines, i.e., achieving \textbf{+30.2\%} on DirectRefusal and \textbf{+21.2\%} on R1-ACT improvement in average reasoning accuracy compared to Vanilla SFT; ii) the degree of reasoning degradation correlates with the extent of distribution shift, suggesting that bridging this gap is central to preserving capabilities. Furthermore, we find that safety alignment in LRMs may primarily function as a mechanism to activate latent knowledge, as a mere \textbf{10} samples are sufficient for activating effective refusal behaviors. These findings not only emphasize the importance of distributional consistency but also provide insights into the activation mechanism of safety in reasoning models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑