Large language models show strong potential for automated code generation, but lack guarantees for correctness, quality, safety, and domain-specific constraints. For instance in robotics, where code generation is increasingly being used for planning and executing actions, awareness of the environment and physical constraints is critical. To facilitate the adaption of code-generating LLMs to diverse requirements, including domain-specific ones, we present a reinforcement learning framework that fine-tunes pre-trained LLMs using proximal policy optimization. Our customizable execution-aware reward formula captures and optimizes syntax, functional correctness, code style, security, and simulator executability. A token-level reward mapping mechanism enables effective credit assignment from execution outcomes to generated tokens. The framework is evaluated on general-purpose code generation (MBPP/MBPP+) and robotic program synthesis (RoboEval). The results show substantial improvements in functional correctness and simulator executability, including an absolute pass@1 increase of 19% on MBPP and a reduction in execution failures by 51% on RoboEval. These findings demonstrate that structured reinforcement learning can effectively align language models to correct program generation and domain-specific requirements.
Balance Between Efficient and Effective Learning: Dense2Sparse Reward Shaping for Robot Manipulation with Environment Uncertainty
Institute of Intelligent Machines, Hefei Institute of Physical Science, CAS(中国科学院合肥物质科学研究院智能机械研究所);Institute of Plasma Physics, Hefei Institute of Physical Science, CAS(中国科学院合肥物质科学研究院等离子体物理研究所)
Rewarding DINO: Predicting Dense Rewards with Vision Foundation Models
Department of Computer Science & Artificial Intelligence, University of Technology Nuremberg(技术大学纽伦堡计算机科学与人工智能系);TUM School of Computation, Information and Technology, Technical University of Munich(慕尼黑技术大学计算、信息与技术学院)
TimeRewarder: Learning Dense Reward from Passive Videos via Frame-wise Temporal Distance
Institute for Interdisciplinary Information Sciences, Tsinghua University, Beijing, China(清华大学交叉信息研究院);Shanghai Qi Zhi Institute(上海启智研究院);Shanghai Jiao Tong University(上海交通大学);University of Pennsylvania(宾夕法尼亚大学)
Grad2Reward: From Sparse Judgment to Dense Rewards for Improving Open-Ended LLM Reasoning
School of Information Science and Technology, ShanghaiTech University(信息科学与技术学院,上海科技大学);State Key Laboratory of General Artificial Intelligence, BIGAI(通用人工智能国家重点实验室,BIGAI)
Adaptable Hindsight Experience Replay for Search-Based Learning
Technical University of Darmstadt(德累斯顿技术大学);Hessian Center for Artificial Intelligence(黑森人工智能中心);German Research Center for Artificial Intelligence(德国人工智能研究中心);Centre for Cognitive Science(认知科学中心);National Research Center for Applied Cybersecurity ATHENE(应用网络安全国家研究中心ATHENE)
本文提出 Adaptable HER 框架,将事后经验回放与 AlphaZero 式搜索学习灵活集成,通过可调整的重新标记目标、策略目标和轨迹选择,在稀疏奖励及方程发现等任务中超越纯监督学习和强化学习。
Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraft
CUHK-SenseTime Joint Laboratory, The Chinese University of Hong Kong(香港中文大学-商汤科技联合实验室,香港中文大学);OpenGVLab, Shanghai AI Laboratory(上海人工智能实验室,OpenGVLab);Tsinghua University(清华大学);SenseTime Research(商汤科技研究院);Shanghai Jiao Tong University(上海交通大学)
An Interpretable Neural Control Network with Adaptable Online Learning for Sample Efficient Robot Locomotion Learning
Vidyasirimedhi Institute of Science and Technology(维达亚希米德科学与技术学院);The Mærsk Mc-Kinney Møller Institute(马士基麦金尼·莫勒研究所);University of Southern Denmark(南方丹麦大学)
Decoding Rewards in Competitive Games: Inverse Game Theory with Entropy Regularization
Department of Electrical and Computer Engineering, Duke University(杜克大学电气与计算机工程系);Department of Statistics and Data Science, University of Pennsylvania(宾夕法尼亚大学统计与数据科学系);Department of Biostatistics and Bioinformatics, Duke University(杜克大学生物统计与生物信息学系);Department of Statistics and Data Science, Yale University(耶鲁大学统计与数据科学系)
Gradients Know What Outcomes Don't: Unlocking Reinforcement Learning for LLM Reasoning with Gradient-Aligned Rewards
Tsinghua University(清华大学);Renmin University of China(中国人民大学);Institute of Information Engineering, CAS(中国科学院信息工程研究所);USTC(中国科学技术大学);The Australian National University(澳大利亚国立大学);Peking University(北京大学);University of Macau(澳门大学)
A Comparison Between Decision Transformers and Traditional Offline Reinforcement Learning Algorithms
School of Mathematics, Statistics and Computer Science(数学、统计与计算机科学学院);University of KwaZulu-Natal(夸祖鲁-纳塔尔大学);Centre for Artificial Intelligence Research (CAIR)(人工智能研究中心)
Dense Process Supervision for Search Agents via Fact Utility Estimation
State Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);Ant Group(蚂蚁集团);National Institute of Healthcare Data Science, Nanjing University(南京大学国家健康医疗数据科学研究院)