评估GRPO和DPO在LLM中的忠实链式推理能力
Evaluating GRPO and DPO for Faithful Chain-of-Thought Reasoning in LLMs
AI总结：
本文评估GRPO和DPO在提升LLM链式推理忠实性方面的性能，发现GRPO在大模型中表现更优，有助于开发更透明可信的推理方法。
摘要：
链式推理（CoT）推理已逐渐成为提升大型语言模型（LLM）问题解决能力的强大技术，尤其适用于需要多步推理的任务。然而，最近的研究表明，CoT解释往往无法反映模型的实际推理过程，因为模型可能会生成连贯但误导性的解释或在不承认外部提示的情况下修改答案。这种差异削弱了基于CoT的方法在安全监督和对齐监控中的可靠性，因为模型可以为错误答案生成合理但具有欺骗性的理由。为了更好地理解这一限制，我们评估了两种优化方法，即群体相对策略优化（GRPO）和直接偏好优化（DPO），在提高CoT忠实性方面的能力。我们的实验表明，GRPO在较大模型中表现优于DPO，Qwen2.5-14B-Instruct模型在所有评估指标上均取得最佳结果。两种方法都表现出模型大小与性能之间的正相关性，但GRPO在提高忠实性指标方面具有更大的潜力，尽管在较小的规模上表现不够稳定。这些结果表明，GRPO为开发更透明和可信的LLM推理方法提供了一个有前景的方向。
英文摘要：
Chain-of-thought (CoT) reasoning has emerged as a powerful technique for improving the problem-solving capabilities of large language models (LLMs), particularly for tasks requiring multi-step reasoning. However, recent studies show that CoT explanations often fail to reflect the model's actual reasoning process, as models may produce coherent yet misleading justifications or modify answers without acknowledging external cues. Such discrepancies undermine the reliability of CoT-based methods for safety supervision and alignment monitoring, as models can generate plausible but deceptive rationales for incorrect answers. To better understand this limitation, we evaluate two optimization methods, Group Relative Policy Optimization (GRPO) and Direct Preference Optimization (DPO), in their ability to improve CoT faithfulness. Our experiments show that GRPO achieves higher performance than DPO in larger models, with the Qwen2.5-14B-Instruct model attaining the best results across all evaluation metrics. Both approaches exhibit positive correlations between model size and performance, but GRPO shows greater potential for improving faithfulness metrics, albeit with less stable behavior at smaller scales. These results suggest that GRPO offers a promising direction for developing more transparent and trustworthy reasoning in LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
