Polaris：基于Syn2Real视觉定位与大语言模型的开放式交互式机器人操作
Polaris: Open-ended Interactive Robotic Manipulation via Syn2Real Visual Grounding and Large Language Models
AI总结：
针对大语言模型缺乏视觉定位制约机器人实体交互的问题，提出融合GPT-4与视觉模型的Polaris框架及Syn2Real位姿估计流程，在桌面操作任务中表现优异且具备泛化潜力。
摘要：
本文研究桌面场景下的开放式交互式机器人操作任务。尽管近期的大语言模型（LLM）提升了机器人对用户指令的理解能力，但这类模型缺乏视觉定位能力，限制了其与物理环境进行实体交互的能力——因为机器人需要在物理工作空间内定位待操作的目标物体。为此，我们提出了名为Polaris的交互式机器人操作框架，该框架结合GPT-4与具备定位能力的视觉模型，整合感知与交互功能。要实现精准操作，这类具备定位能力的视觉模型必须输出目标物体的详细位姿，而非仅识别图像中属于目标物体的像素。因此，我们提出了一种新颖的合成到真实（Syn2Real）位姿估计流程。该流程使用渲染的合成数据进行训练，随后迁移至真实世界的操作任务中。真实场景下的性能验证了所提流程的有效性，也凸显了其向更通用类别拓展的潜力。此外，真实机器人实验表明，该框架在抓取及执行多种操作任务上表现出色，具备泛化至桌面以外场景的潜力。更多信息与视频结果可访问：https://star-uu-wang.github.io/Polaris/
英文摘要：
This paper investigates the task of the open-ended interactive robotic manipulation on table-top scenarios. While recent Large Language Models (LLMs) enhance robots' comprehension of user instructions, their lack of visual grounding constrains their ability to physically interact with the environment. This is because the robot needs to locate the target object for manipulation within the physical workspace. To this end, we introduce an interactive robotic manipulation framework called Polaris, which integrates perception and interaction by utilizing GPT-4 alongside grounded vision models. For precise manipulation, it is essential that such grounded vision models produce detailed object pose for the target object, rather than merely identifying pixels belonging to them in the image. Consequently, we propose a novel Synthetic-to-Real (Syn2Real) pose estimation pipeline. This pipeline utilizes rendered synthetic data for training and is then transferred to real-world manipulation tasks. The real-world performance demonstrates the efficacy of our proposed pipeline and underscores its potential for extension to more general categories. Moreover, real-robot experiments have showcased the impressive performance of our framework in grasping and executing multiple manipulation tasks. This indicates its potential to generalize to scenarios beyond the tabletop. More information and video results are available here: https://star-uu-wang.github.io/Polaris/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
