Search-augmented large language models (LLMs) trained with reinforcement learning (RL) have achieved strong results on open-domain question answering (QA), but training still remains a significant challenge. The optimization is often unstable due to sparse rewards and difficult credit assignments across reasoning and tool calls. To address this, we introduce Turn-Level Information Potential Reward Shaping (TIPS), a simple framework that assigns dense, turn-level rewards to each reasoning + tool-call segment based on the increased likelihood of the correct answer under a teacher model. By leveraging the potential-based reward shaping, TIPS offers fine-grained and policy-invariant guidance that overcomes the limitations of outcome-only optimization. Evaluated on seven QA benchmarks, TIPS consistently outperforms GRPO/PPO baselines and substantially improves training stability. For instance, with a Qwen-2.5 7B Instruct model, TIPS improves the average Exact Match score by 11.8% and F1 by 13.6% relative to PPO. Our results demonstrate that turn-level information-potential reward shaping provides an effective and general solution to sparse-reward credit assignment for multi-turn LLM reasoning.
Policy-Invariant Reward Shaping from LLM Feedback: A Framework for Hybrid RL Agents
AI Research and Innovation Nexus for Africa (AIRINA Labs)（非洲AI研究与创新中心（AIRINA实验室））；Sefako Makgatho Health Sciences University (SMU)（塞法科·马加托健康科学大学）；African Center for Advanced Studies (ACAS)（非洲高级研究中心）；African Institute for Mathematical Sciences（非洲数学科学学院）
Environment Shaping in Reinforcement Learning using State Abstraction
EPFL（洛桑联邦理工学院）；MPI-SWS（马克斯·普朗克软件系统研究所）
强化学习（RL）代理所面临的核心挑战之一是在具有稀疏和噪声反馈信号的大状态空间的环境中有效地学习（接近）最优策略。在实际应用中，具有额外领域知识的专家可以通过\emph{shaping the environment}帮助加速学习过程，即，使学习环境更加友好。文献中流行的范例是\emph{potential-based reward shaping}，其中环境的奖励函数使用潜在函数用额外的局部奖励来增强。然而，基于潜力的奖励塑造的适用性在以下设置中受到限制：（i）状态空间非常大，并且计算适当的潜力函数具有挑战性;（ii）反馈信号是嘈杂的，并且即使具有整形的奖励，代理也可能陷入局部最优;以及（iii）单独改变奖励是不够的，并且有效的整形需要改变动态。我们解决这些限制的潜在的成形方法，并提出了一个新的框架\emph{environment shaping using state abstraction}。我们的主要思想是压缩环境的大状态空间与嘈杂的信号到一个抽象的空间，并使用这种抽象在创建更平滑，更有效的反馈信号的代理。我们研究了我们的抽象为基础的环境塑造的理论基础，并表明，代理人的政策在塑造的环境中保存在原始环境中接近最优的行为。
AMARIS: A Memory-Augmented Rubric Improvement System for Rubric-Based Reinforcement Learning
The University of Texas at Dallas（德克萨斯大学达拉斯分校）；Adobe Inc.（Adobe公司）；Department of Computer Science, University of California, Santa Barbara（加州大学圣芭芭拉分校计算机科学系）
LACONIC: Length-Aware Constrained Reinforcement Learning for LLM
University of California, Los Angeles（加州大学洛杉矶分校）；Salesforce AI Research（Salesforce AI研究院）；National University of Singapore（新加坡国立大学）；University of Alberta（阿尔伯塔大学）
