迈向LLM中的安全推理：用于策略嵌入CoT数据创建的AI智能体审议
Towards Safety Reasoning in LLMs: AI-agentic Deliberation for Policy-embedded CoT Data Creation
AI总结：
针对LLM安全推理范式实施中高质量策略嵌入CoT数据集创建的挑战，提出AIDSAFE多智能体迭代审议数据生成方案，并辅以信念增强构建偏好数据，显著提升开源LLM的安全泛化与越狱鲁棒性。
摘要：
安全推理是一种近期出现的范式，LLm在响应生成前对安全策略进行推理，从而缓解现有安全措施（如过度拒绝和越狱漏洞）的局限。然而，实施这一范式具有挑战性，因为创建高质量策略嵌入思维链（CoT）数据集是一个资源密集型过程，同时必须确保推理保持准确且无幻觉或策略冲突。为此，我们提出AIDSAFE：安全推理的智能体迭代审议，这是一种利用多智能体审议来迭代扩展展对安全策略推理的新数据生成方案。AIDSAFE中的数据精炼阶段通过消除重复、冗余和欺骗性思维来确保高质量输出。AIDSAFE生成的CoT为基于监督微调（SFT）的安全训练提供了坚实基础。此外，为了解决对齐阶段（如DPO训练）偏好数据的需求，我们引入了一种补充方案，该方案利用信念增强来创建不同的选定和拒绝的CoT样本。我们的评估表明，AIDSAFE生成的CoT实现了卓越的策略遵从和推理质量。因此，我们表明在这些CoT上微调开源LLM可以显著提高安全泛化和越狱鲁棒性，同时保持可接受的实用性和过度拒绝准确度。AIDSAFE生成的CoT数据集可在此处找到：https://huggingface.co/datasets/AmazonScience/AIDSAFE
英文摘要：
Safety reasoning is a recent paradigm where LLMs reason over safety policies before generating responses, thereby mitigating limitations in existing safety measures such as over-refusal and jailbreak vulnerabilities. However, implementing this paradigm is challenging due to the resource-intensive process of creating high-quality policy-embedded chain-of-thought (CoT) datasets while ensuring reasoning remains accurate and free from hallucinations or policy conflicts. To tackle this, we propose AIDSAFE: Agentic Iterative Deliberation for Safety Reasoning, a novel data generation recipe that leverages multi-agent deliberation to iteratively expand reasoning on safety policies. A data refiner stage in AIDSAFE ensures high-quality outputs by eliminating repetitive, redundant, and deceptive thoughts. AIDSAFE-generated CoTs provide a strong foundation for supervised fine-tuning (SFT)-based safety training. Additionally, to address the need of preference data in alignment stages, such as DPO training, we introduce a supplemental recipe that uses belief augmentation to create distinct selected and rejected CoT samples. Our evaluations demonstrate that AIDSAFE-generated CoTs achieve superior policy adherence and reasoning quality. Consequently, we show that fine-tuning open-source LLMs on these CoTs can significantly improve safety generalization and jailbreak robustness while maintaining acceptable utility and over-refusal accuracy. AIDSAFE-generated CoT datasets can be found here: https://huggingface.co/datasets/AmazonScience/AIDSAFE
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
