将互联网规模视觉语言模型蒸馏到具身智能体中
Distilling Internet-Scale Vision-Language Models into Embodied Agents
AI总结：
该研究提出结合模型蒸馏与后见经验回放，用互联网规模预训练视觉语言模型监督具身智能体，通过提示控制监督信号，实现新物体交互与抽象类别教学，为具身智能体的语言锚定提供了新路径。
摘要：
遵循指令的智能体必须将语言锚定到其观测和动作空间中。学习语言锚定极具挑战性，通常需要特定领域的工程设计或大量人类交互数据。为应对这一挑战，我们提出使用预训练视觉语言模型（VLM）来监督具身智能体。我们结合了模型蒸馏和后见经验回放（HER）的思路，利用VLM回溯生成描述智能体行为的语言。简单的提示工程使我们能够控制监督信号，在3D渲染环境中教会智能体基于物体名称（如planes）或特征（如颜色）与新物体交互。少样本提示可让我们教授抽象类别归属，包括已有类别（食物与玩具）和特设类别（对物体的任意偏好）。我们的工作提出了一种使用互联网规模VLM的新型有效方法，将此类模型习得的通用语言锚定能力重新用于向具身智能体传授任务相关的锚定能力。
英文摘要：
Instruction-following agents must ground language into their observation and action spaces. Learning to ground language is challenging, typically requiring domain-specific engineering or large quantities of human interaction data. To address this challenge, we propose using pretrained vision-language models (VLMs) to supervise embodied agents. We combine ideas from model distillation and hindsight experience replay (HER), using a VLM to retroactively generate language describing the agent's behavior. Simple prompting allows us to control the supervision signal, teaching an agent to interact with novel objects based on their names (e.g., planes) or their features (e.g., colors) in a 3D rendered environment. Fewshot prompting lets us teach abstract category membership, including pre-existing categories (food vs toys) and ad-hoc ones (arbitrary preferences over objects). Our work outlines a new and effective way to use internet-scale VLMs, repurposing the generic language grounding acquired by such models to teach task-relevant groundings to embodied agents.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
