arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.18756cs.LGcs.AIcs.CL
收藏

复杂损失函数是否必要用于教LLM进行推理?

Are complicated loss functions necessary for teaching LLMs to reason?

DEIB, Politecnico di Milano(米兰理工学院DEIB学院)

Gabriele Carrino, Andrea Sassella, Nicolo Brunello, Federico Toschi, Mark James Carman

AI总结:

本文通过分析GRPO发现负反馈对训练至关重要,PPO风格约束非必需,提出简化版RGRA在数学基准上表现更优。

摘要:

近期大语言模型(LLMs)的进步凸显了训练后技术对推理和数学能力的重要性。组相对策略优化(GRPO)通过结合组相对优势估计、PPO风格裁剪和KL正则化在该领域显示出潜力。然而,其复杂性引发了是否所有组件对培养推理行为都是必要的问题。我们系统分析GRPO并发现两点关键结论:(1)仅在基准之上训练动作时,纳入负反馈是必要的;(2)PPO风格约束如策略比率裁剪并非改进数学推理或性能所必需。基于这些见解,我们提出组相对优势的REINFORCE(RGRA),一种简化版本,保留组相对优势估计但去除PPO风格裁剪和策略比率项。在标准数学基准上的实验表明,RGRA有潜力比GRPO表现更优。我们的结果表明,更简单的基于REINFORCE的方法可以有效提升LLMs的推理能力,提供比GRPO更透明和高效的替代方案。

英文摘要:

Recent advances in large language models (LLMs) highlight the importance of post training techniques for improving reasoning and mathematical ability. Group Relative Policy Optimization (GRPO) has shown promise in this domain by combining group relative advantage estimation, PPO style clipping, and KL regularization. However, its complexity raises the question of whether all components are necessary for fostering reasoning behaviors. We conduct a systematic analysis of GRPO and identify two key findings: (1) incorporating negative feedback is essential training solely on actions above a baseline limits learning; and (2) PPO style constraints, such as policy ratio clipping, are not required to improve mathematical reasoning or performance. Building on these insights, we propose REINFORCE with Group Relative Advantage (RGRA), a simplified variant that retains group relative advantage estimation but removes PPO style clipping and policy ratio terms. Experiments across standard mathematical benchmarks indicate that RGRA has the potential to achieve stronger performance than GRPO. Our results suggest that simpler REINFORCE based approaches can effectively enhance reasoning in LLMs, offering a more transparent and efficient alternative to GRPO.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑