语言模型辅助的双层规划用于从互联网视频中学习奖励
Language-Model-Assisted Bi-Level Programming for Reward Learning from Internet Videos
AI总结：
提出语言模型辅助的双层规划框架，通过VLM和LLM协作直接从互联网视频学习奖励，无需复杂数据准备，实现高效奖励设计与复杂行为合成。
摘要：
从示范中学习，特别是从人类和动物等生物专家的示范中学习，经常遇到显著的数据获取挑战。虽然最近的方法利用互联网视频进行学习，但它们需要复杂的、特定任务的流水线来提取和重定向运动数据供智能体使用。在这项工作中，我们引入了一个语言模型辅助的双层规划框架，使强化学习智能体能够直接从互联网视频中学习其奖励，绕过专门的数据准备。该框架包括两个层级：上层是视觉语言模型（VLM）通过将学习者的行为与专家视频进行比较来提供反馈，下层是大型语言模型（LLM）将此反馈转化为奖励更新。VLM和LLM在这个双层框架内协作，使用“链式法则”方法推导出奖励学习的有效搜索方向。我们验证了从YouTube视频进行奖励学习的方法，结果表明所提出的方法能够从生物智能体的专家视频中实现高效的奖励设计，用于复杂行为合成。
英文摘要：
Learning from Demonstrations, particularly from biological experts like humans and animals, often encounters significant data acquisition challenges. While recent approaches leverage internet videos for learning, they require complex, task-specific pipelines to extract and retarget motion data for the agent. In this work, we introduce a language-model-assisted bi-level programming framework that enables a reinforcement learning agent to directly learn its reward from internet videos, bypassing dedicated data preparation. The framework includes two levels: an upper level where a vision-language model (VLM) provides feedback by comparing the learner's behavior with expert videos, and a lower level where a large language model (LLM) translates this feedback into reward updates. The VLM and LLM collaborate within this bi-level framework, using a "chain rule" approach to derive a valid search direction for reward learning. We validate the method for reward learning from YouTube videos, and the results have shown that the proposed method enables efficient reward design from expert videos of biological agents for complex behavior synthesis.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
