强化学习（RL）的环境转移模型存在固有不确定性，因此需要在探索与利用之间实现精妙平衡。这种平衡对于优化计算资源、准确估计智能体的期望奖励至关重要。在奖励稀疏的场景（例如机器人控制系统）中，实现这种平衡尤其具有挑战性。然而，鉴于许多环境拥有丰富的先验知识，在这类场景中从零开始学习可能是冗余的。为解决这一问题，我们提出了语言模型引导奖励调优（Language Model Guided reward Tuning，LMGT）——一种全新的样本高效框架。LMGT利用大语言模型（LLMs）中蕴含的丰富先验知识，以及它们处理非标准数据形式（如维基教程）的能力。通过采用LLM引导的奖励偏移策略，LMGT能够巧妙平衡探索与利用，从而引导智能体的探索行为并提升样本效率。我们已在多种强化学习任务中对LMGT进行了严格评估，并在具身机器人环境Housekeep中开展了测试。结果表明，LMGT的性能始终优于基线方法。此外，研究结果显示我们的框架能够大幅降低强化学习训练阶段所需的计算资源。
英文摘要：
The inherent uncertainty in the environmental transition model of Reinforcement Learning (RL) necessitates a delicate balance between exploration and exploitation. This balance is crucial for optimizing computational resources to accurately estimate expected rewards for the agent. In scenarios with sparse rewards, such as robotic control systems, achieving this balance is particularly challenging. However, given that many environments possess extensive prior knowledge, learning from the ground up in such contexts may be redundant. To address this issue, we propose Language Model Guided reward Tuning (LMGT), a novel, sample-efficient framework. LMGT leverages the comprehensive prior knowledge embedded in Large Language Models (LLMs) and their proficiency in processing non-standard data forms, such as wiki tutorials. By utilizing LLM-guided reward shifts, LMGT adeptly balances exploration and exploitation, thereby guiding the agent's exploratory behavior and enhancing sample efficiency. We have rigorously evaluated LMGT across various RL tasks and evaluated it in the embodied robotic environment Housekeep. Our results demonstrate that LMGT consistently outperforms baseline methods. Furthermore, the findings suggest that our framework can substantially reduce the computational resources required during the RL training phase.
Co-Evolution of Policy and Internal Reward for Language Agents
McGill University（麦吉尔大学）；McMaster University（麦克马斯特大学）；The University of Hong Kong（香港大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Peking University（北京大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；DeepWisdom（深度智慧）；Université de Montréal（蒙特利尔大学）；Mila（米拉研究所）
SocraticPO: Policy Optimization via Interactive Guidance
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China（认知智能国家重点实验室，中国科学技术大学）；iFLYTEK AI Research (Central China), iFLYTEK Co., Ltd（iFLYTEK中央中国AI研究院，iFLYTEK公司）
Reinforcement Learning with Curriculum-inspired Adaptive Direct Policy Guidance for Truck Dispatching
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Learning Diverse Policies with Soft Self-Generated Guidance
School of Mathematical Sciences, Beihang University（北京航空航天大学数学科学学院）；Institute of Artificial Intelligence, Beihang University（北京航空航天大学人工智能研究院）；Zhongguancun Laboratory（中关村实验室）
Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoning
Key Laboratory of Network Data Science（网络数据科学实验室）；Technology, Institute of Computing Technology, Chinese Academy of Sciences（技术，计算技术研究所，中国科学院）；State Key Laboratory of AI Safety（人工智能安全国家重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；University of California, Merced（加州大学梅尔塞德斯分校）
Dual-Granularity Contrastive Reward via Generated Episodic Guidance for Efficient Embodied RL
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，自动化研究所，中国科学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（人工智能学院，中国科学院大学）
Hybrid-AIRL: Enhancing Inverse Reinforcement Learning with Supervised Expert Guidance
Artificial Intelligence Lab — Vrije Universiteit Brussel（自由大学布鲁塞尔人工智能实验室）；College of Science and Engineering — University of Galway（盖尔威大学科学与工程学院）；Family Medicine and Population Health — University of Antwerp（安特卫普大学家庭医学与人口健康）
Policy Optimization with Smooth Guidance Learned from State-Only Demonstrations
School of Mathematical Sciences, Beihang University（北京航空航天大学数学科学学院）；Institute of Artificial Intelligence, Beihang University（北京航空航天大学人工智能研究院）；Zhongguancun Laboratory（中关村实验室）；Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing（未来区块链与隐私计算北京高精尖创新中心）
PRPO: Aligning Process Reward with Outcome Reward in Policy Optimization
Shanghai University（上海大学）；University of Science and Technology of China（中国科学技术大学）；Shanghai Jiaotong University（上海交通大学）；Artificial Intelligence Incubation and Innovation Institute, Fudan University（复旦大学人工智能孵化与创新研究院）；Shanghai Academy of AI for Science（上海人工智能科学研究院）
Eliminating Inductive Bias in Reward Models with Information-Theoretic Guidance
Qwen Large Model Application Team, Alibaba（阿里巴巴大模型应用团队）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Research Institute of Big Data（深圳大数据研究院）
KGRGRL: A User's Permission Reasoning Method Based on Knowledge Graph Reward Guidance Reinforcement Learning
Command and Control Engineering College, Army Engineering University of PLA（中国人民解放军陆军工程大学指挥控制工程学院）；troop of PLA（中国人民解放军31436部队）；Academy of Military Science（军事科学院）
MARVL: Multi-Stage Guidance for Robotic Manipulation via Vision-Language Models
School of Intelligent Science and Technology, Nanjing University, China（南京大学智能科学与技术学院）；National Key Laboratory for Novel Software Technology, School of Artificial Intelligence, Nanjing University, China（南京大学新型软件技术国家实验室，人工智能学院）；School of Artificial Intelligence, Nanjing University, China（南京大学人工智能学院）；MACS Lab, University of Washington（华盛顿大学MACS实验室）
Cheems: A Practical Guidance for Building and Evaluating Chinese Reward Models from Scratch
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所信息处理实验室）；University of Chinese Academy of Sciences（中国科学院大学）
MVR: Multi-view Video Reward Shaping for Reinforcement Learning
School of Intelligence Science and Technology, Peking University（北京大学智能科学与技术学院）；State Key Laboratory of General Artificial Intelligence, BIGAI（通用人工智能国家重点实验室）
