arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.14092cs.ROcs.AI
收藏

使用大型语言模型学习物理技能的奖励

Learning Reward for Physical Skills using Large Language Model

Yuwei Zeng, Yiqing Xu

AI总结:

本文提出利用环境反馈从大型语言模型中提取任务知识,通过LLM提出奖励函数特征并迭代自对齐更新参数,解决物理技能奖励函数学习的挑战。

摘要:

由于技能范围广泛、状态和动作空间的高维性以及细微的感官反馈,为物理技能学习奖励函数具有挑战性。这些任务的复杂性使得获取专家演示数据既昂贵又耗时。大型语言模型(LLMs)包含有价值的任务相关知识,可以辅助学习这些奖励函数。然而,直接应用LLMs提出奖励函数有其局限性,例如数值不稳定性和无法结合环境反馈。我们旨在利用环境反馈从LLMs中提取任务知识,为物理技能创建高效的奖励函数。我们的方法包含两个部分。我们首先使用LLM提出奖励函数的特征和参数化。接下来,我们通过迭代自对齐过程更新所提出奖励函数的参数。具体而言,该过程基于新观测最小化LLM与我们学习到的奖励函数之间的排序不一致性。我们通过在三个模拟物理技能学习任务上测试我们的方法进行了验证,展示了对我们设计选择的有效支持。

英文摘要:

Learning reward functions for physical skills are challenging due to the vast spectrum of skills, the high-dimensionality of state and action space, and nuanced sensory feedback. The complexity of these tasks makes acquiring expert demonstration data both costly and time-consuming. Large Language Models (LLMs) contain valuable task-related knowledge that can aid in learning these reward functions. However, the direct application of LLMs for proposing reward functions has its limitations such as numerical instability and inability to incorporate the environment feedback. We aim to extract task knowledge from LLMs using environment feedback to create efficient reward functions for physical skills. Our approach consists of two components. We first use the LLM to propose features and parameterization of the reward function. Next, we update the parameters of this proposed reward function through an iterative self-alignment process. In particular, this process minimizes the ranking inconsistency between the LLM and our learned reward functions based on the new observations. We validated our method by testing it on three simulated physical skill learning tasks, demonstrating effective support for our design choices.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑