arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.16546cs.AI
收藏

基于通用领域知识的高效行为克隆

Sample-Efficient Behavior Cloning Using General Domain Knowledge

Carnegie Mellon University(卡内基梅隆大学)

Feiyu Zhu, Jean Oh, Reid Simmons

AI总结:

本文提出KIM方法,通过整合大语言模型的编码能力与专家领域知识,提升行为克隆的样本效率和泛化能力。

摘要:

行为克隆在许多序列决策任务中通过学习专家示范而取得了成功,但它们往往非常样本不高效,并且无法泛化到未见场景。为解决这些问题,引入通用领域知识,使策略能够聚焦于本质特征,并通过应用该知识泛化到未见状态。尽管这种知识容易从专家处获得,但将其与单个示例的学习结合困难,因为神经网络缺乏语义结构,且特征工程耗时。为了使策略能够同时学习通用知识和具体示范轨迹,我们利用大语言模型的编码能力,基于用自然语言表达的专家领域知识构建策略结构,并通过示范调整策略参数。我们将此方法称为知识引导模型(KIM),因为结构反映了专家知识的语义。在月球着陆器和赛车任务的实验中,我们的方法仅需5个示范即可学会任务,并对动作噪声具有鲁棒性,优于没有领域知识的基线模型。这表明,借助大语言模型,我们能够将领域知识融入策略结构,提高行为克隆的样本效率。

英文摘要:

Behavior cloning has shown success in many sequential decision-making tasks by learning from expert demonstrations, yet they can be very sample inefficient and fail to generalize to unseen scenarios. One approach to these problems is to introduce general domain knowledge, such that the policy can focus on the essential features and may generalize to unseen states by applying that knowledge. Although this knowledge is easy to acquire from the experts, it is hard to be combined with learning from individual examples due to the lack of semantic structure in neural networks and the time-consuming nature of feature engineering. To enable learning from both general knowledge and specific demonstration trajectories, we use a large language model's coding capability to instantiate a policy structure based on expert domain knowledge expressed in natural language and tune the parameters in the policy with demonstrations. We name this approach the Knowledge Informed Model (KIM) as the structure reflects the semantics of expert knowledge. In our experiments with lunar lander and car racing tasks, our approach learns to solve the tasks with as few as 5 demonstrations and is robust to action noise, outperforming the baseline model without domain knowledge. This indicates that with the help of large language models, we can incorporate domain knowledge into the structure of the policy, increasing sample efficiency for behavior cloning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑