ManipGPT：大型视觉模型的可供性分割是否足以实现铰接物体操作？
ManipGPT: Is Affordance Segmentation by Large Vision Models Enough for Articulated Object Manipulation?
AI总结：
本文提出ManipGPT框架，利用大型预训练ViT和9.9k图像数据集进行部件级可供性分割，结合阻抗适应策略，实现铰接物体在模拟和现实中的有效操作。
摘要：
视觉可操作可供性已成为机器人学中一种变革性方法，专注于在操作之前感知交互区域。传统方法依赖像素采样来识别成功的交互样本，或处理点云以进行可供性映射。然而，这些方法计算密集，且难以适应多样化和动态的环境。本文介绍了ManipGPT，一个利用大型预训练视觉变换器（ViT）预测铰接物体最优交互区域的框架。我们创建了一个包含9.9k张模拟和真实图像的数据集，以弥合视觉模拟到真实的差距，并增强现实世界的适用性。通过在这个小数据集上微调视觉变换器，我们显著改善了部件级可供性分割，将模型的上下文内分割能力适应到机器人操作场景中。这通过生成部件级可供性掩码，并结合阻抗适应策略，实现了在模拟和现实环境中的有效操作，充分消除了对复杂数据集或感知系统的需求。
英文摘要：
Visual actionable affordance has emerged as a transformative approach in robotics, focusing on perceiving interaction areas prior to manipulation. Traditional methods rely on pixel sampling to identify successful interaction samples or processing pointclouds for affordance mapping. However, these approaches are computationally intensive and struggle to adapt to diverse and dynamic environments. This paper introduces ManipGPT, a framework designed to predict optimal interaction areas for articulated objects using a large pre-trained vision transformer (ViT). We create a dataset of 9.9k simulated and real images to bridge the visual sim-to-real gap and enhance real-world applicability. By fine-tuning the vision transformer on this small dataset, we significantly improve part-level affordance segmentation, adapting the model's in-context segmentation capabilities to robot manipulation scenarios. This enables effective manipulation across simulated and real-world environments by generating part-level affordance masks, paired with an impedance adaptation policy, sufficiently eliminating the need for complex datasets or perception systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
