从优化角度纠正大语言模型的思维
Rectifying LLM Thought from Lens of Optimization
AI总结：
本文通过优化视角分析LLM推理过程，提出RePro方法改进推理性能，减少过度思考等亚优行为。
摘要：
近年来，大语言模型（LLM）的进步主要得益于其涌现的推理能力，尤其是长链式思维（CoT）提示，能够深入探索和推理。尽管如此，长CoT LLMs常表现出亚优的推理行为，如过度思考和过长的推理链，影响性能。本文通过优化视角分析推理过程，将CoT视为梯度下降过程，每个推理步骤为问题解决的更新。基于此视角，我们引入RePro（纠正过程级奖励），一种在训练后优化LLM推理的新方法。RePro定义了一个替代目标函数来评估CoT背后的优化过程，利用双评分机制量化其强度和稳定性。这些评分被整合成一个过程级奖励，无缝集成到具有可验证奖励的强化学习（RLVR）流程中以优化LLM。在多个强化学习算法和多种LLM上进行的广泛实验，评估了数学、科学和编码等基准，证明RePro一致地提高了推理性能并减轻了亚优推理行为。
英文摘要：
Recent advancements in large language models (LLMs) have been driven by their emergent reasoning capabilities, particularly through long chain-of-thought (CoT) prompting, which enables thorough exploration and deliberation. Despite these advances, long-CoT LLMs often exhibit suboptimal reasoning behaviors, such as overthinking and excessively protracted reasoning chains, which can impair performance. In this paper, we analyze reasoning processes through an optimization lens, framing CoT as a gradient descent procedure where each reasoning step constitutes an update toward problem resolution. Building on this perspective, we introduce RePro (Rectifying Process-level Reward), a novel approach to refine LLM reasoning during post-training. RePro defines a surrogate objective function to assess the optimization process underlying CoT, utilizing a dual scoring mechanism to quantify its intensity and stability. These scores are aggregated into a composite process-level reward, seamlessly integrated into reinforcement learning with verifiable rewards (RLVR) pipelines to optimize LLMs. Extensive experiments across multiple reinforcement learning algorithms and diverse LLMs, evaluated on benchmarks spanning mathematics, science, and coding, demonstrate that RePro consistently enhances reasoning performance and mitigates suboptimal reasoning behaviors.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
