SDGO：用于大语言模型一致安全性的自判别引导优化
SDGO: Self-Discrimination-Guided Optimization for Consistent Safety in Large Language Models
AI总结：
针对大语言模型存在的安全不一致性问题，提出自判别引导优化（SDGO）强化学习框架，利用模型自身判别能力作为奖励信号迭代优化生成安全性，无需额外数据或外部模型，显著提升对越狱攻击的防御能力并保持通用任务性能。
摘要：
大语言模型（LLMs）在各类自然语言处理任务中表现出色，但仍易受诱导生成有害内容的越狱攻击。本文揭示了一个关键的安全不一致性问题：LLMs作为判别器识别有害请求的效果，优于其作为生成器防御这些请求的效果。这一见解启发我们探索对齐模型固有的判别与生成能力。为此，我们提出SDGO（自判别引导优化），这是一种强化学习框架，利用模型自身的判别能力作为奖励信号，通过迭代自我改进增强生成安全性。我们的方法在训练阶段不需要任何额外的标注数据或外部模型。大量实验表明，与基于提示和基于训练的基线相比，SDGO显著提高了模型的安全性，同时在通用基准上保持了有用性。通过对齐LLMs的判别和生成能力，SDGO对分布外（OOD）越狱攻击具有稳健性能。这种对齐实现了两种能力之间更紧密的耦合，使模型的生成能力仅需少量判别样本即可进一步增强。我们的代码和数据集可在https://github.com/NJUNLP/SDGO获取。
英文摘要：
Large Language Models (LLMs) excel at various natural language processing tasks but remain vulnerable to jailbreaking attacks that induce harmful content generation. In this paper, we reveal a critical safety inconsistency: LLMs can more effectively identify harmful requests as discriminators than defend against them as generators. This insight inspires us to explore aligning the model's inherent discrimination and generation capabilities. To this end, we propose SDGO (Self-Discrimination-Guided Optimization), a reinforcement learning framework that leverages the model's own discrimination capabilities as a reward signal to enhance generation safety through iterative self-improvement. Our method does not require any additional annotated data or external models during the training phase. Extensive experiments demonstrate that SDGO significantly improves model safety compared to both prompt-based and training-based baselines while maintaining helpfulness on general benchmarks. By aligning LLMs' discrimination and generation capabilities, SDGO brings robust performance against out-of-distribution (OOD) jailbreaking attacks. This alignment achieves tighter coupling between these two capabilities, enabling the model's generation capability to be further enhanced with only a small amount of discriminative samples. Our code and datasets are available at https://github.com/NJUNLP/SDGO.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
