arXiv 2506.172192025-06-25cs.LGcs.AI

没有免费午餐：重新思考LLM推理的内部反馈

No Free Lunch: Rethinking Internal Feedback for LLM Reasoning

Zhongguancun Academy（中关村学院）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Peking University（北京大学）；Zhongguancun Institute of Artificial Intelligence（中关村人工智能研究院）

Yanzhi Zhang, Zhaoxi Zhang, Haoxiang Guan, Yilin Cheng, Yitong Duan, Chen Wang, Yue Wang, Shuxin Zheng, Jiyan He

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文研究了内部反馈强化学习（RLIF）在LLM推理后训练中的表现，发现其初期可提升基础模型性能，但随训练进行性能反而下降，且对指令微调模型改进有限。

摘要：

强化学习已成为一种强大的范式，用于对大型语言模型（LLM）进行后训练以提升推理能力。像人类反馈强化学习（RLHF）和可验证奖励强化学习（RLVR）这样的方法已经显示出强大的结果，但它们需要广泛的外部监督。我们研究了一类替代方法，即内部反馈强化学习（RLIF），它仅依赖内在的模型衍生信号而不是外部奖励。特别是，我们利用了无监督奖励代理，如token级别的熵、轨迹级别的熵和自我确定性。我们的理论分析表明这些内部目标是部分等价的，我们在具有挑战性的数学推理基准上实证评估了各种RLIF策略。实验结果表明，RLIF在训练的初始阶段可以提升基础LLM的推理性能，在这些任务上匹配或超越RLVR技术。然而，随着训练的进行，性能会下降，甚至低于训练前的模型。此外，我们发现RLIF对指令微调模型的改进微乎其微，这表明一旦LLM已经过指令微调，内在反馈的收益就会递减。我们通过混合模型权重进一步分析了这一局限性，并解释了RLIF训练行为的原因，为将内部反馈信号整合到LLM训练中提供了实用指南。我们希望我们对内部反馈的分析能为LLM后训练提供更有原则和有效的策略。

英文摘要：

Reinforcement learning has emerged as a powerful paradigm for post-training large language models (LLMs) to improve reasoning. Approaches like Reinforcement Learning from Human Feedback (RLHF) and Reinforcement Learning with Verifiable Rewards (RLVR) have shown strong results, but they require extensive external supervision. We investigate an alternative class of methods, Reinforcement Learning from Internal Feedback (RLIF), which relies solely on intrinsic model-derived signals instead of external rewards. In particular, we leverage unsupervised reward proxies such as token-level entropy, trajectory-level entropy, and self-certainty. Our theoretical analysis shows these internal objectives are partially equivalent, and we empirically evaluate various RLIF strategies on challenging math reasoning benchmarks. Experimental results demonstrate that RLIF can boost the reasoning performance of base LLMs at the beginning phase of the training, matching or surpassing RLVR techniques on these tasks. However, when training progresses, performance degrades even below the model before training. Moreover, we find that RLIF yields little improvement for instruction-tuned models, indicating diminishing returns of intrinsic feedback once an LLM is already instruction-tuned. We further analyze this limitation by mixing model weights and explain the reason of RLIF's training behaviors, providing practical guidelines for integrating internal feedback signals into LLM training. We hope our analysis of internal feedback will inform more principled and effective strategies for LLM post-training.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。