Oyster-II：大语言模型中用于建设性安全对齐的强化学习
Oyster-II: Reinforcement Learning for Constructive Safety Alignment in Large Language Models
AI总结：
研究大语言模型安全等问题，指出传统策略不足。基于Oyster-I范式，提出Oyster-II框架，采用强化学习结合零强化学习范式，在多基准测试中安全维度表现出色。
摘要：
大语言模型在各种应用中展现出卓越能力，但确保其安全性、有用性和可信度仍是挑战。传统拒绝导向对齐策略存在问题。基于Oyster-I开创的建设性安全范式，识别出其基于监督微调方案的局限性，提出Oyster-II框架，在广泛基准测试中安全维度全面超越Qwen3-14B和Oyster-I。
英文摘要：
Large language models (LLMs) have demonstrated remarkable capabilities across diverse applications, yet ensuring their simultaneous safety, helpfulness, and trustworthiness remains a persistent challenge. Conventional refusal-oriented alignment strategies mitigate harmful content generation but systematically fail to serve legitimate user needs, often withholding information that could safely and constructively address the underlying intent of sensitive queries. Building upon the constructive safety paradigm pioneered by Oyster-I, which moves beyond blanket refusal toward thoughtful, response-oriented safety alignment, we identify two critical limitations of its Supervised Fine-Tuning (SFT)-based scheme: insufficient safety generalization to out-of-distribution scenarios and a phenomenon we term safety chain-of-thought (CoT) over-generalization, wherein safety-oriented reasoning patterns are excessively applied to benign queries, degrading helpfulness and user experience. To address these limitations, we propose Oyster-II, a reinforcement learning (RL)-based constructive safety alignment framework that adopts a Zero-RL paradigm combined with a multi-stage reinforcement learning strategy.Evaluated across extensive benchmarks, Oyster-II comprehensively surpasses both Qwen3-14B and its predecessor Oyster-I on safety dimensions, achieving cross-scale performance comparable to Qwen3-Max and Qwen3.5-397B.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
