推理是否保持对齐？关于大型推理模型的可信度研究
Does Reasoning Preserve Alignment? On the Trustworthiness of Large Reasoning Models
AI总结：
研究通过监督微调、强化学习和蒸馏生成的推理模型在安全、偏见、隐私等六个可信度维度上是否保持对齐，发现推理模型常出现对齐退化，如毒性增加、刻板印象加剧等。
摘要：
经过指令微调的LLM越来越多地通过后训练转化为推理模型，以提高多步任务性能。这种转化通常针对推理准确性进行优化，而没有明确保留指令微调模型的对齐行为，如安全拒绝、避免偏见和隐私保护。我们提出疑问：这种转化是否保持对齐？我们通过可信度审计研究这个问题，并发现默认情况下它并不保持行为。为了系统分析，我们比较了通过监督微调、基于RL的后训练和蒸馏产生的推理模型，与匹配的指令微调基线在六个可信度维度上的表现：安全性、毒性、刻板印象与偏见、机器伦理、隐私和分布外鲁棒性。我们观察到推理模型通常在推理基准上有所改进，但表现出对齐退化，包括毒性增加、刻板印象加剧、拒绝校准错误和上下文隐私泄露。这些退化与从指令微调基线的行为漂移一致，通过KL散度测量。总体而言，我们的结果指向更广泛的结论：可信度指标对于评估推理模型至关重要，并且应与推理能力的提升一起报告。
英文摘要：
Instruction-tuned LLMs are increasingly converted into reasoning models through post-training to improve multi-step task performance. This conversion is usually optimized for reasoning accuracy, without explicitly preserving the alignment behavior of the instruction-tuned model, such as safe refusal, bias avoidance, and privacy protection. We ask: does this conversion preserve alignment? We study this question through a trustworthiness audit and find that it is not behavior-preserving by default. For a systematic analysis, we compare reasoning models produced via supervised fine-tuning, RL-based post-training, and distillation against matched instruction-tuned baselines across six trustworthiness dimensions: safety, toxicity, stereotyping and bias, machine ethics, privacy, and out-of-distribution robustness. We observe that reasoning models often improve on reasoning benchmarks but exhibit alignment regressions, including increased toxicity, amplified stereotyping, miscalibrated refusal, and contextual privacy leakage. These regressions are consistent with behavioral drift from the instruction-tuned baseline, measured by KL divergence. Overall, our results point to the broader conclusion that trustworthiness metrics are essential for evaluating reasoning models and should be reported alongside gains in reasoning capability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
