通过自演化评分标准强化链式推理
Reinforcing Chain-of-Thought Reasoning with Self-Evolving Rubrics
AI总结：
RLCER通过自演化评分标准提升链式推理的奖励机制，无需人工标注即可实现更高效的推理性能。
摘要：
尽管链式推理（CoT）在大语言模型（LLM）推理中起着关键作用，但直接奖励它具有挑战性：训练奖励模型需要大量的人工标注工作，而静态的RMs难以应对不断演变的CoT分布和奖励黑客问题。这些挑战促使我们寻求一种无需人工标注且能逐步演化的自主CoT奖励方法。受最近的自演化训练方法启发，我们提出了RLCER（通过自演化评分标准的强化学习），它通过自提出和自演化评分标准来奖励CoT，从而增强以结果为导向的RLVR。我们证明，自提出和自演化评分标准即使在没有结果奖励的情况下也能提供可靠的CoT监督信号，使RLCER优于以结果为导向的RLVR。此外，当这些自提出评分标准用作提示中的提示信息时，进一步提升了推理时的性能。
英文摘要：
Despite chain-of-thought (CoT) playing crucial roles in LLM reasoning, directly rewarding it is difficult: training a reward model demands heavy human labeling efforts, and static RMs struggle with evolving CoT distributions and reward hacking. These challenges motivate us to seek an autonomous CoT rewarding approach that requires no human annotation efforts and can evolve gradually. Inspired by recent self-evolving training methods, we propose \textbf{RLCER} (\textbf{R}einforcement \textbf{L}earning with \textbf{C}oT Supervision via Self-\textbf{E}volving \textbf{R}ubrics), which enhances the outcome-centric RLVR by rewarding CoTs with self-proposed and self-evolving rubrics. We show that self-proposed and self-evolving rubrics provide reliable CoT supervision signals even without outcome rewards, enabling RLCER to outperform outcome-centric RLVR. Moreover, when used as in-prompt hints, these self-proposed rubrics further improve inference-time performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
