ImageManip：基于图像的机器人操作与可供性引导的下一视角选择
ImageManip: Image-based Robotic Manipulation with Affordance-guided Next View Selection
AI总结：
本文提出ImageManip，一种基于RGB图像的机器人操作框架，通过多视角捕捉和深度推断，利用可供性引导的下一视角选择，以低成本实现高效三维物体操作。
摘要：
在未来的家庭辅助机器人领域，三维关节物体的操作对于使机器人能够与环境交互至关重要。许多现有研究利用三维点云作为操作策略的主要输入。然而，这种方法由于数据稀疏以及获取点云数据的高昂成本而面临挑战，这可能限制其实用性。相比之下，RGB图像通过成本低廉的设备提供高分辨率观测，但缺乏空间三维几何信息。为了克服这些限制，我们提出了一种新颖的基于图像的机器人操作框架。该框架旨在捕捉目标物体的多个视角，并推断深度信息以补充其几何结构。最初，系统采用眼在手上的RGB相机捕捉目标物体的整体视图。它预测初始深度图和粗略的可供性图。可供性图指示物体上可操作区域，并作为选择后续视角的约束。基于全局视觉先验，我们自适应地确定最佳下一视角，以详细观察潜在操作成功区域。我们利用几何一致性融合视图，从而获得更精细的深度图和更精确的可供性图，用于机器人操作决策。通过与采用点云或RGB图像作为输入的先前工作进行对比，我们证明了我们方法的有效性和实用性。在项目网页（https://sites.google.com/view/imagemanip）上，真实世界实验进一步凸显了我们方法在实际部署中的潜力。
英文摘要：
In the realm of future home-assistant robots, 3D articulated object manipulation is essential for enabling robots to interact with their environment. Many existing studies make use of 3D point clouds as the primary input for manipulation policies. However, this approach encounters challenges due to data sparsity and the significant cost associated with acquiring point cloud data, which can limit its practicality. In contrast, RGB images offer high-resolution observations using cost effective devices but lack spatial 3D geometric information. To overcome these limitations, we present a novel image-based robotic manipulation framework. This framework is designed to capture multiple perspectives of the target object and infer depth information to complement its geometry. Initially, the system employs an eye-on-hand RGB camera to capture an overall view of the target object. It predicts the initial depth map and a coarse affordance map. The affordance map indicates actionable areas on the object and serves as a constraint for selecting subsequent viewpoints. Based on the global visual prior, we adaptively identify the optimal next viewpoint for a detailed observation of the potential manipulation success area. We leverage geometric consistency to fuse the views, resulting in a refined depth map and a more precise affordance map for robot manipulation decisions. By comparing with prior works that adopt point clouds or RGB images as inputs, we demonstrate the effectiveness and practicality of our method. In the project webpage (https://sites.google.com/view/imagemanip), real world experiments further highlight the potential of our method for practical deployment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
