发表机构
Harbin Institute of Technology (Shenzhen); Leiden University（哈尔滨工业大学（深圳）; 莱顿大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对现有安全对齐易受越狱攻击且过度拒绝的问题，提出SSRFT框架，通过安全角色问答数据集内化安全价值观，实现比标准SFT更鲁棒、更可泛化的安全对齐，同时减少过度拒绝并保持通用能力。
AI 中文摘要
大型语言模型（LLMs）已展现出卓越的能力，但仍易受到旨在诱导有害或不安全输出的越狱攻击。现有的安全对齐方法，包括监督微调（SFT）和基于人类反馈的强化学习（RLHF），通常需要大量针对特定攻击的监督和计算资源，同时仍易受浅层安全对齐和过度拒绝的影响。为应对这些挑战，我们提出了SSRFT（监督安全角色微调），这是首个将安全对齐重新定义为预定义安全角色内化的框架。SSRFT从心理测量问题、有限的越狱提示和安全角色描述中构建了一个安全角色问答（SRQA）数据集。角色一致的响应被合成、验证并扩展到多样场景，使模型能够内化以安全为导向的价值观和原则，而非显式的拒绝模式。在多个基础模型和指令模型上的实验表明，SSRFT比标准SFT实现了更鲁棒且更可泛化的安全对齐。SSRFT对预填充攻击表现出显著更强的鲁棒性，对未见过的越狱领域具有更好的泛化能力，同时减少了对良性查询的过度拒绝，并保持了模型的通用能力。这些结果确立了安全角色内化作为以拒绝为中心的安全对齐的有效替代方案。警告：本文包含有害和有毒语言的示例。
英文摘要
Large Language Models (LLMs) have achieved remarkable capabilities but remain vulnerable to jailbreak attacks that elicit harmful or unsafe outputs. Existing safety alignment approaches, including Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF), often require substantial attack-specific supervision and computational resources, while remaining susceptible to shallow safety alignment and over-refusal. To address these challenges, we introduce SSRFT(Supervised Safe-Role Fine-Tuning), the first framework that reformulates safety alignment as the internalization of a predefined safe role. SSRFT constructs a Safe-Role Question-Answer (SRQA) dataset from psychometric questions, limited jailbreak prompts, and a safe-role description. Role-consistent responses are synthesized, validated, and expanded into diverse scenarios, enabling models to internalize safety-oriented values and principles rather than explicit refusal patterns. Experiments across multiple Base and Instruct models show that SSRFT achieves more robust and generalizable safety alignment than standard SFT. SSRFT shows substantially greater robustness to prefilling attacks and better generalization to unseen jailbreak domains, while reducing over-refusal on benign queries and preserving the model's general capabilities. These results establish safe-role internalization as an effective alternative to refusal-centric safety alignment. Warning: This paper contains examples of harmful and toxic language.
Comments27 pages,7 figures, under review
