基于记忆增强师生学习的提示响应式物体抓取
Prompt-responsive Object Retrieval with Memory-augmented Student-Teacher Learning
AI总结：
本文提出记忆增强师生学习框架，将SAM 2可提示基础模型与强化学习结合，利用不完美检测的时间序列进行隐式状态估计，使机器人能够根据用户提示在杂乱场景中完成灵巧抓取任务。
摘要：
构建对输入提示做出响应的模型代表了机器学习的一次变革性转变。这一范式对机器人问题具有重大潜力，例如在杂乱环境中的定向操作。在本工作中，我们提出了一种新颖的方法，将可提示的基础模型与强化学习（RL）相结合，使机器人能够以提示响应的方式执行灵巧操作任务。现有方法难以将高层命令与细粒度的灵巧控制联系起来。我们通过一个记忆增强的师生学习框架来弥补这一差距。我们使用Segment-Anything 2（SAM 2）模型作为感知骨干，从用户提示中推断感兴趣的目标物体。虽然检测结果并不完美，但它们的时间序列为记忆增强模型进行隐式状态估计提供了丰富的信息。我们的方法成功学习到了提示响应策略，并在从杂乱场景中抓取物体的任务中得到了验证。视频和代码可在 https://memory-student-teacher.github.io 获取。
英文摘要：
Building models responsive to input prompts represents a transformative shift in machine learning. This paradigm holds significant potential for robotics problems, such as targeted manipulation amidst clutter. In this work, we present a novel approach to combine promptable foundation models with reinforcement learning (RL), enabling robots to perform dexterous manipulation tasks in a prompt-responsive manner. Existing methods struggle to link high-level commands with fine-grained dexterous control. We address this gap with a memory-augmented student-teacher learning framework. We use the Segment-Anything 2 (SAM 2) model as a perception backbone to infer an object of interest from user prompts. While detections are imperfect, their temporal sequence provides rich information for implicit state estimation by memory-augmented models. Our approach successfully learns prompt-responsive policies, demonstrated in picking objects from cluttered scenes. Videos and code are available at https://memory-student-teacher.github.io
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
