Recent advances in large language models (LLMs) highlight the importance of post training techniques for improving reasoning and mathematical ability. Group Relative Policy Optimization (GRPO) has shown promise in this domain by combining group relative advantage estimation, PPO style clipping, and KL regularization. However, its complexity raises the question of whether all components are necessary for fostering reasoning behaviors. We conduct a systematic analysis of GRPO and identify two key findings: (1) incorporating negative feedback is essential training solely on actions above a baseline limits learning; and (2) PPO style constraints, such as policy ratio clipping, are not required to improve mathematical reasoning or performance. Building on these insights, we propose REINFORCE with Group Relative Advantage (RGRA), a simplified variant that retains group relative advantage estimation but removes PPO style clipping and policy ratio terms. Experiments across standard mathematical benchmarks indicate that RGRA has the potential to achieve stronger performance than GRPO. Our results suggest that simpler REINFORCE based approaches can effectively enhance reasoning in LLMs, offering a more transparent and efficient alternative to GRPO.
Teaching Algorithmic Reasoning via In-context Learning
Mila(米拉研究所);Université de Montreal(蒙特利尔大学);Google Research(谷歌研究院)
大语言模型(LLM)通过扩展模型和数据大小,显示出越来越多的上下文学习能力。尽管取得了这些进展,LLM仍然无法解决算法推理问题。虽然为最终答案提供了一个基本原理,导致了多步推理问题的进一步改进,但Anil et al. 2022表明,即使是简单的算法推理任务,如奇偶校验,也远未解决。在这项工作中,我们确定并研究了成功教授算法推理的四个关键阶段:(1)将算法制定为技能,(2)同时教授多种技能(技能积累),(3)教授如何组合技能(技能组合)和(4)教授如何使用技能作为工具。我们表明,它是可以教算法推理LLMs通过上下文学习,我们称之为算法提示。我们评估我们的方法在各种算术和定量推理任务,并表现出显着的提升性能比现有的提示技术。特别是,对于长奇偶校验、加法、乘法和减法,与最佳可用基线相比,我们分别实现了约10倍、9倍、5倍和2倍的误差减少。
Teaching LLM to Reason: Reinforcement Learning from Algorithmic Problems without Code
University of Science and Technology of China(中国科学技术大学);Alibaba Group(阿里巴巴集团);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
Bayesian Teaching Enables Probabilistic Reasoning in Large Language Models
MIT(麻省理工学院);Meta;Google(谷歌公司);DeepMind University of British Columbia(不列颠哥伦比亚大学深度思维学院);Vector Institute(向量研究所);New York University(纽约大学);Google Research(谷歌研究)
Teaching LLMs According to Their Aptitude: Adaptive Reasoning for Mathematical Problem Solving
The Hong Kong University of Science and Technology(香港科学与技术大学);Huawei Noah’s Ark Lab(华为诺亚实验室);Zhejiang University(浙江大学);Peking University(北京大学);The Hong Kong Polytechnic University(香港理工大学);University of Surrey(萨里大学)
Teaching LLMs for Step-Level Automatic Math Correction via Reinforcement Learning
School of Computer Science and Technology, East China Normal University(华东师范大学计算机科学与技术学院);Lab of AI for Education, East China Normal University(华东师范大学教育人工智能实验室)
Tailored Teaching with Balanced Difficulty: Elevating Reasoning in Multimodal Chain-of-Thought via Prompt Curriculum
Nanjing University of Aeronautics and Astronautics(南京航空航天大学);Hunan Vanguard Group Corporation Co., Ltd.(湖南 Vanguard集团有限公司);Huaneng Information Technology Co., Ltd.(华能信息技术有限公司);Tsinghua University(清华大学);Zhejiang University(浙江大学);PowerChina Huadong Engineering Co., Ltd.(中国电建华东工程有限公司)
Acting Less is Reasoning More! Teaching Model to Act Efficiently
The Chinese University of Hong Kong(香港中文大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴娜-香槟分校);Princeton University(普林斯顿大学);Sun Yat-sen University(中山大学);Hong Kong University of Science and Technology(香港科学理工大学)
Teaching AI Stepwise Diagnostic Reasoning with Report-Guided Chain-of-Thought Learning
Fujian University of Technology(福建工程学院);Fujian Provincial Key Laboratory of Big Data Mining and Applications(福建省大数据挖掘与应用重点实验室);Key Laboratory of Biomedical Imaging Science and System, Chinese Academy of Sciences(生物医学成像科学与系统重点实验室,中国科学院)
SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thought
SIGS, Tsinghua University(清华大学信息科学与技术学院);Southern University of Science and Technology(南方科技大学);Guangdong Laboratory of AI and Digital Economy (SZ)(广东人工智能与数字经济实验室);The Hong Kong University of Science and Technology(香港科技大学);Peking University(北京大学)
Teaching Code LLMs to Reason with Intermediate Formal Specifications
FPT Software AI Center(FPT软件人工智能中心);University of Texas at Dallas(德克萨斯大学达拉斯分校);Hanoi Univ. of Science and Tech.(河内科学技术大学);Texas, USA(得克萨斯州,美国);Hanoi, Vietnam(河内,越南)
The Challenge of Teaching Reasoning to LLMs Without RL or Distillation
Nvidia Corporation(Nvidia公司);Institute for Artificial Intelligence Research(人工智能研究机构);Faculty of Technical Sciences, University of Novi Sad(诺维萨德大学技术科学学院);Department of Mathematics and Informatics, Faculty of Sciences, University of Novi Sad(诺维萨德大学科学学院数学与信息学系);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Chicago(芝加哥大学);Georgia Institute of Technology(佐治亚理工学院);Department of Computer and Information Science, School of Engineering and Applied Science, University of Pennsylvania(宾夕法尼亚大学计算机与信息科学系);Massachusetts Institute of Technology(麻省理工学院)
Legal Syllogism Prompting: Teaching Large Language Models for Legal Judgment Prediction
Peking University Law School(北京大学法学院);Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);PKU-WUHAN Institute for Artificial Intelligence(北大-武汉人工智能研究院)
S$^2$R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learning
Tencent(腾讯);Tsinghua University(清华大学);The University of Hong Kong(香港大学);Fudan University(复旦大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
Teaching the Way, Not the Answer: Privileged Tutoring Distillation for Multimodal Policy Optimization
Tianjin University(天津大学);Beijing Institute of Technology(北京理工大学);Singapore Management University(新加坡国立大学);University of Chinese Academy of Sciences(中国科学院大学);Xiaomi Inc(小米公司)