arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.07162cs.ROcs.AI
收藏

基于大语言模型自对齐的机器人技能奖励学习

Learning Reward for Robot Skills Using Large Language Models via Self-Alignment

National University of Singapore(新加坡国立大学);The University of Hong Kong(香港大学)

Yuwei Zeng, Yao Mu, Lin Shao

AI总结:

针对机器人奖励函数学习瓶颈,提出基于LLM自对齐的无人工奖励学习方法,通过LLM生成奖励特征与参数、迭代自对齐更新参数,在2个仿真环境9项任务上提升训练效果与效率,且GPT token消耗显著低于变异法。

摘要:

奖励函数学习仍是让机器人掌握广泛技能库的瓶颈。大语言模型(LLM)蕴含宝贵的任务相关知识,有望助力奖励函数学习。然而,其提出的奖励函数可能不够精确,因此效果不佳,需要结合环境信息进一步落地。我们提出了一种无需人类参与即可更高效学习奖励的方法。该方法包含两个组件:首先利用LLM提出奖励的特征与参数化方式,随后通过迭代自对齐过程更新参数。具体而言,该过程基于执行反馈,最小化LLM与所学奖励函数之间的排序不一致性。我们在2个仿真环境中的9项任务上验证了该方法。结果表明,该方法在训练效果与效率上均实现了持续提升,同时与基于变异的替代方法相比,消耗的GPT token数量显著更少。

英文摘要:

Learning reward functions remains the bottleneck to equip a robot with a broad repertoire of skills. Large Language Models (LLM) contain valuable task-related knowledge that can potentially aid in the learning of reward functions. However, the proposed reward function can be imprecise, thus ineffective which requires to be further grounded with environment information. We proposed a method to learn rewards more efficiently in the absence of humans. Our approach consists of two components: We first use the LLM to propose features and parameterization of the reward, then update the parameters through an iterative self-alignment process. In particular, the process minimizes the ranking inconsistency between the LLM and the learnt reward functions based on the execution feedback. The method was validated on 9 tasks across 2 simulation environments. It demonstrates a consistent improvement over training efficacy and efficiency, meanwhile consuming significantly fewer GPT tokens compared to the alternative mutation-based method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑