Learning reward functions remains the bottleneck to equip a robot with a broad repertoire of skills. Large Language Models (LLM) contain valuable task-related knowledge that can potentially aid in the learning of reward functions. However, the proposed reward function can be imprecise, thus ineffective which requires to be further grounded with environment information. We proposed a method to learn rewards more efficiently in the absence of humans. Our approach consists of two components: We first use the LLM to propose features and parameterization of the reward, then update the parameters through an iterative self-alignment process. In particular, the process minimizes the ranking inconsistency between the LLM and the learnt reward functions based on the execution feedback. The method was validated on 9 tasks across 2 simulation environments. It demonstrates a consistent improvement over training efficacy and efficiency, meanwhile consuming significantly fewer GPT tokens compared to the alternative mutation-based method.
From Relevance to Execution Utility: Reward-Aware Dynamic Execution Gating for Skill-Based LLM Agents
Tongji University(同济大学);The University of Sydney(悉尼大学);University of Science and Technology of China(中国科学技术大学);Nankai University(南开大学);Johns Hopkins University(约翰斯·霍普金斯大学);City University of Hong Kong(香港城市大学);University of Surrey(萨里大学)
SelfEvoSkill: Paired Execution Audits for Skill Revision without External Outcome Supervision
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS(中国科学院计算技术研究所人工智能安全国家重点实验室);University of Chinese Academy of Sciences(中国科学院大学);Tongyi Lab, Alibaba Group(阿里巴巴集团通义实验室)
Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment
Chinese University of Hong Kong(香港中文大学);Microsoft Research Asia(微软亚洲研究院);Peking University(北京大学);Hong Kong University of Science and Technology(香港科技大学)
Co-Evolving LLM Decision and Skill Bank Agents for Long-Horizon Tasks
University of Maryland(马里兰大学);University of Southern California(南加州大学);Good Start Labs(Good Start实验室);Independent Researcher(独立研究者);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
R$^2$ Flow: Recursive Self-Improvement via Recursive Skill Evolution
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Fudan University(复旦大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Oxford(牛津大学);National University of Singapore(新加坡国立大学)
SkillNEO: Non-Elitist Evolutionary Optimization of Agent Skills
University of Illinois at Chicago(伊利诺伊大学芝加哥分校);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);University of California, San Diego(加利福尼亚大学圣迭戈分校);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);University of Texas at Arlington(德克萨斯大学阿灵顿分校)
Learning While Acting: A Skill-Enhanced Test-Time Co-Evolution Framework for Online Lifelong Learning Agents
School of Computer Science and Technology, East China Normal University(东华大学计算机科学与技术学院);Shanghai AI Laboratory(上海人工智能实验室);Software Engineering Institute, East China Normal University(东华大学软件工程学院)
Evaluating and Improving Cultural Awareness of Reward Models for LLM Alignment
Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China(计算与智能研究所,哈尔滨工业大学,深圳,中国);Peng Cheng Laboratory, Shenzhen, China(鹏城实验室,深圳,中国)
Does LLM Alignment Really Need Diversity? An Empirical Study of Adapting RLVR Methods for Moral Reasoning
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);Microsoft Research(微软研究院);University of Michigan(密歇根大学);Shanghai Jiao Tong University(上海交通大学);CUHKSZ(香港中文大学(深圳));THU(清华大学)
ComSD: Balancing Behavioral Quality and Diversity in Unsupervised Skill Discovery
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Nanjing Institute of Software Technology(南京软件技术研究院)
MA-ROESL: Motion-aware Rapid Reward Optimization for Efficient Robot Skill Learning from Single Videos
College of Information Science and Technology, Eastern Institute of Technology(信息科学与技术学院,东部技术学院);School of Computer Science and Technology, The University of Science and Technology of China(计算机科学与技术学院,中国科学技术大学);Department of Computing, The Hong Kong Polytechnic University(计算学院,香港理工大学)
Causal Reward World Models: Zero-shot Reward Design for Automated Skill Generation
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Intelligent Manufacturing Institute, HFUT(合肥工业大学智能制造研究院);School of Electrical Engineering and Automation, Anhui University(安徽大学电气工程与自动化学院);School of Artificial Intelligence, China University of Mining and Technology (Beijing)(中国矿业大学(北京)人工智能学院);National Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institution of Automation, Chinese Academy of Sciences(中国科学院自动化研究所复杂系统认知与决策智能全国重点实验室)
Robotic Skill Diversification via Active Mutation of Reward Functions in Reinforcement Learning During a Liquid Pouring Task
Cognitive Robotics, Delft University of Technology(德鲁特理工大学认知机器人学);Department of Mechanical Engineering, Politecnico di Milano(米兰理工学院机械工程系);Istituto Dalle Molle di studi sull’intelligenza artificiale (IDSIA USI-SUPSI), Scuola universitaria professionale della Svizzera italiana, DTI(智能人工智能研究所(IDSIA USI-SUPSI),瑞士意大利大学专业学院,DTI)
GROVE: A Generalized Reward for Learning Open-Vocabulary Physical Skill
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);State Key Laboratory of General Artificial Intelligence, BIGAI(北京通用人工智能研究院 通用人工智能国家重点实验室);School of Control Science and Engineering, Shandong University(山东大学控制科学与工程学院);Department of Automation, Tsinghua University(清华大学自动化系)
Diverse Skill Discovery for Quadruped Robots via Unsupervised Learning
College of Intelligent Robotics and Advanced Manufacturing(智能机器人与先进制造学院);Fudan University(复旦大学);College of Foreign Language(外国语言学院);The University of Shanghai for Science and Technology(上海科技大学)