Reinforcement learning (RL) has been effective for post-training autoregressive (AR) language models, but extending these methods to diffusion language models (DLMs) is challenging due to intractable sequence-level likelihoods. Existing approaches therefore rely on surrogate likelihoods or heuristic approximations, which can introduce bias and obscure the sequential structure of denoising. We formulate diffusion-based sequence generation as a finite-horizon Markov decision process over the denoising trajectory and derive an exact, unbiased policy gradient that decomposes over denoising steps and is expressed in terms of intermediate advantages, without requiring explicit evaluation of the sequence likelihood. To obtain a practical and compute-efficient estimator, we (i) select denoising steps for policy updates via an entropy-guided approximation bound, and (ii) estimate intermediate advantages using a one-step denoising reward naturally provided by the diffusion model, avoiding costly multi-step rollouts. Experiments on coding and logical reasoning benchmarks demonstrate state-of-the-art results, with strong competitive performance on mathematical reasoning, outperforming existing RL post-training approaches for DLMs. Code is available at https://github.com/vishnutez/egspo-dllm-rl.
Rubric-Guided Process Reward for Stepwise Model Routing
University of Science and Technology of China（中国科学技术大学）；Southeast University（东南大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）
Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPO
Department of Industrial Engineering and Operations Research（工业工程与运营管理系）；Department of Mathematics（数学系）；Columbia University（哥伦比亚大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Stern School of Business, New York University（纽约大学斯特恩商学院）
School of Software Technology, Zhejiang University（浙江大学软件学院）；Zhejiang Key Laboratory of Digital-Intelligence Service Technology, Zhejiang University（浙江大学数字智能服务技术重点实验室）；ZWSOFT Co., Ltd.（ZWSOFT公司）
Self-Guided Function Calling in Large Language Models via Stepwise Experience Recall
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences（认知与决策智能复杂系统重点实验室，自动化研究所，中国科学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（人工智能学院，中国科学院大学）；Nanjing University of Information Science & Technology（南京信息工程大学）；Institute of Computing Technology, Chinese Academy of Sciences（计算技术研究所，中国科学院）
$\mathcal{S}^2$IT: Stepwise Syntax Integration Tuning for Large Language Models in Aspect Sentiment Quad Prediction
School of Computer Science, Guangdong University of Technology（广东技术大学计算机科学学院）；Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（广东人工智能与数字经济实验室（深圳））；Peng Cheng Laboratory（鹏城实验室）；College of Science, Shantou University（汕头大学理学院）
Bridging Modal Isolation in Interleaved Thinking: Supervising Modality Transitions via Stepwise Reinforcement
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shanghai Jiaotong University（上海交通大学）；Zhejiang University（浙江大学）；University of Chinese Academy of Sciences（中国科学院大学）
STRIDE: Learnable Stepwise Language Feedback for LLM Reasoning
Generative AI Lab, College of Computing and Data Science, Nanyang Technological University, Singapore（生成式人工智能实验室，计算与数据科学学院，南洋理工大学，新加坡）；Tongyi Lab, Alibaba Group（通义实验室，阿里巴巴集团）
Perceiving the World: Question-guided Reinforcement Learning for Text-based Games
University of Technology Sydney（悉尼科技大学）；Eindhoven University of Technology（埃因霍温理工大学）；King’s College London（伦敦国王学院）；A*STAR Centre for Frontier AI Research（新加坡科技研究局前沿人工智能研究中心）
我们提供了一个加速强化学习（RL）算法的框架，由领域知识或离线数据构造的算法。Tabula Rasa RL算法需要与顺序决策任务的范围相匹配的环境交互或计算。使用我们的框架，我们展示了如何启发式引导RL诱导一个更短的地平线子问题，可证明解决了原来的任务。我们的框架可以被看作是一个基于视界的正则化，用于在有限的相互作用预算下控制RL中的偏差和方差。在理论方面，我们描述了一个很好的启发式和RL加速的影响的性质。特别是，我们引入了一个可改进的启发式，启发式，允许RL代理推断超出其先验知识的新概念。在经验方面，我们实例化我们的框架，以加速几个国家的最先进的算法在模拟机器人控制任务和程序生成的游戏。我们的框架补充了关于热启动RL的丰富文献，包括专家演示或探索性数据集，并介绍了将先验知识注入RL的原则性方法。
Stepwise Reasoning Enhancement for LLMs via External Subgraph Generation
School of Information Science and Engineering, Chongqing Jiaotong University（重庆交通大学信息科学与工程学院）；School of Computer Science and Technology, Chongqing University of Posts and Telecommunications（重庆邮电大学计算机科学与技术学院）
FlowQ: Energy-Guided Flow Policies for Offline Reinforcement Learning
Machine Learning Research Lab, Volkswagen Group（大众集团机器学习研究实验室）；Technical University of Munich（慕尼黑技术大学）；Foundation Robotics Labs, Munich（慕尼黑基础机器人实验室）；Eötvös Loránd University（劳尔·艾尔多迪大学）
TGPO: Tree-Guided Preference Optimization for Robust Web Agent Reinforcement Learning
East China Normal University（东华大学）；Alibaba Group（阿里巴巴集团）；University of Electronic Science and Technology of China（电子科技大学）；Wuhan University（武汉大学）；Sun Yat-sen University（中山大学）
本文提出树引导偏好优化（TGPO）框架，通过树结构轨迹表示、过程奖励模型和动态加权机制，解决 Web Agent 强化学习中的信用分配错位、标注成本高和奖励稀疏问题，并在两个数据集上取得更高成功率和更少冗余步骤。
Deep Reinforcement Learning Guided Improvement Heuristic for Job Shop Scheduling
Nanyang Technological University（南洋理工大学）；Singapore Management University（新加坡管理大学）；Institute of Marine Science and Technology, Shandong University（山东大学海洋科学与技术研究院）；Eindhoven University of Technology（埃因霍温理工大学）
