Learning reward functions for physical skills are challenging due to the vast spectrum of skills, the high-dimensionality of state and action space, and nuanced sensory feedback. The complexity of these tasks makes acquiring expert demonstration data both costly and time-consuming. Large Language Models (LLMs) contain valuable task-related knowledge that can aid in learning these reward functions. However, the direct application of LLMs for proposing reward functions has its limitations such as numerical instability and inability to incorporate the environment feedback. We aim to extract task knowledge from LLMs using environment feedback to create efficient reward functions for physical skills. Our approach consists of two components. We first use the LLM to propose features and parameterization of the reward function. Next, we update the parameters of this proposed reward function through an iterative self-alignment process. In particular, this process minimizes the ranking inconsistency between the LLM and our learned reward functions based on the new observations. We validated our method by testing it on three simulated physical skill learning tasks, demonstrating effective support for our design choices.
From Relevance to Execution Utility: Reward-Aware Dynamic Execution Gating for Skill-Based LLM Agents
Tongji University(同济大学);The University of Sydney(悉尼大学);University of Science and Technology of China(中国科学技术大学);Nankai University(南开大学);Johns Hopkins University(约翰斯·霍普金斯大学);City University of Hong Kong(香港城市大学);University of Surrey(萨里大学)
SelfEvoSkill: Paired Execution Audits for Skill Revision without External Outcome Supervision
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS(中国科学院计算技术研究所人工智能安全国家重点实验室);University of Chinese Academy of Sciences(中国科学院大学);Tongyi Lab, Alibaba Group(阿里巴巴集团通义实验室)
Co-Evolving LLM Decision and Skill Bank Agents for Long-Horizon Tasks
University of Maryland(马里兰大学);University of Southern California(南加州大学);Good Start Labs(Good Start实验室);Independent Researcher(独立研究者);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
R$^2$ Flow: Recursive Self-Improvement via Recursive Skill Evolution
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Fudan University(复旦大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Oxford(牛津大学);National University of Singapore(新加坡国立大学)
SkillNEO: Non-Elitist Evolutionary Optimization of Agent Skills
University of Illinois at Chicago(伊利诺伊大学芝加哥分校);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);University of California, San Diego(加利福尼亚大学圣迭戈分校);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);University of Texas at Arlington(德克萨斯大学阿灵顿分校)
Learning While Acting: A Skill-Enhanced Test-Time Co-Evolution Framework for Online Lifelong Learning Agents
School of Computer Science and Technology, East China Normal University(东华大学计算机科学与技术学院);Shanghai AI Laboratory(上海人工智能实验室);Software Engineering Institute, East China Normal University(东华大学软件工程学院)
ComSD: Balancing Behavioral Quality and Diversity in Unsupervised Skill Discovery
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Nanjing Institute of Software Technology(南京软件技术研究院)
MA-ROESL: Motion-aware Rapid Reward Optimization for Efficient Robot Skill Learning from Single Videos
College of Information Science and Technology, Eastern Institute of Technology(信息科学与技术学院,东部技术学院);School of Computer Science and Technology, The University of Science and Technology of China(计算机科学与技术学院,中国科学技术大学);Department of Computing, The Hong Kong Polytechnic University(计算学院,香港理工大学)
Causal Reward World Models: Zero-shot Reward Design for Automated Skill Generation
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Intelligent Manufacturing Institute, HFUT(合肥工业大学智能制造研究院);School of Electrical Engineering and Automation, Anhui University(安徽大学电气工程与自动化学院);School of Artificial Intelligence, China University of Mining and Technology (Beijing)(中国矿业大学(北京)人工智能学院);National Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institution of Automation, Chinese Academy of Sciences(中国科学院自动化研究所复杂系统认知与决策智能全国重点实验室)
Robotic Skill Diversification via Active Mutation of Reward Functions in Reinforcement Learning During a Liquid Pouring Task
Cognitive Robotics, Delft University of Technology(德鲁特理工大学认知机器人学);Department of Mechanical Engineering, Politecnico di Milano(米兰理工学院机械工程系);Istituto Dalle Molle di studi sull’intelligenza artificiale (IDSIA USI-SUPSI), Scuola universitaria professionale della Svizzera italiana, DTI(智能人工智能研究所(IDSIA USI-SUPSI),瑞士意大利大学专业学院,DTI)
GROVE: A Generalized Reward for Learning Open-Vocabulary Physical Skill
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);State Key Laboratory of General Artificial Intelligence, BIGAI(北京通用人工智能研究院 通用人工智能国家重点实验室);School of Control Science and Engineering, Shandong University(山东大学控制科学与工程学院);Department of Automation, Tsinghua University(清华大学自动化系)
Diverse Skill Discovery for Quadruped Robots via Unsupervised Learning
College of Intelligent Robotics and Advanced Manufacturing(智能机器人与先进制造学院);Fudan University(复旦大学);College of Foreign Language(外国语言学院);The University of Shanghai for Science and Technology(上海科技大学)
SkillMimic: Learning Basketball Interaction Skills from Demonstrations
Hong Kong University of Science and Technology(香港科技大学);Unitree Robotics(宇树机器人);Peking University Shenzhen Graduate School(北京大学深圳研究生院);Tsinghua University(清华大学);International Digital Economy Academy(国际数字经济学院);Tencent(腾讯);Carnegie Mellon University(卡内基梅隆大学)