PhyGrasp：利用物理信息大语言多模态模型实现机器人抓取泛化
PhyGrasp: Generalizing Robotic Grasping with Physics-informed Large Multimodal Models
AI总结：
PhyGrasp通过融合自然语言与3D点云的多模态大模型，将物理常识推理引入机器人抓取，在长尾场景下显著提升了抓取成功率。
摘要：
机器人抓取是机器人功能的基本方面，定义了机器人如何与物体交互。尽管取得了重大进展，但其对反直觉或长尾场景（例如具有不常见材料或形状的物体）的泛化能力仍然是一个挑战。相比之下，人类可以轻松地运用其直觉物理来熟练地抓取并高效地改变抓取方式，即使对于他们从未见过的物体也是如此。本工作深入探讨将这种物理常识推理注入机器人操作中。我们引入了PhyGrasp，这是一种多模态大模型，利用来自两种模态的输入：自然语言和3D点云，通过一个桥接模块无缝集成。语言模态展现出关于不同物理属性对抓取影响的鲁棒推理能力，而3D模态则理解物体形状和部件。凭借这两种能力，PhyGrasp能够准确评估物体部件的物理属性并确定最佳抓取姿态。此外，该模型的语言理解能力使其能够解释人类指令，生成符合人类偏好的抓取姿态。为了训练PhyGrasp，我们构建了一个数据集PhyPartNet，包含195K个具有不同物理属性和人类偏好的物体实例，以及它们对应的语言描述。在仿真和真实机器人上进行的大量实验表明，PhyGrasp实现了最先进的性能，特别是在长尾情况下，例如在GraspNet上的成功率提高了约10%。项目页面：https://sites.google.com/view/phygrasp
英文摘要：
Robotic grasping is a fundamental aspect of robot functionality, defining how robots interact with objects. Despite substantial progress, its generalizability to counter-intuitive or long-tailed scenarios, such as objects with uncommon materials or shapes, remains a challenge. In contrast, humans can easily apply their intuitive physics to grasp skillfully and change grasps efficiently, even for objects they have never seen before. This work delves into infusing such physical commonsense reasoning into robotic manipulation. We introduce PhyGrasp, a multimodal large model that leverages inputs from two modalities: natural language and 3D point clouds, seamlessly integrated through a bridge module. The language modality exhibits robust reasoning capabilities concerning the impacts of diverse physical properties on grasping, while the 3D modality comprehends object shapes and parts. With these two capabilities, PhyGrasp is able to accurately assess the physical properties of object parts and determine optimal grasping poses. Additionally, the model's language comprehension enables human instruction interpretation, generating grasping poses that align with human preferences. To train PhyGrasp, we construct a dataset PhyPartNet with 195K object instances with varying physical properties and human preferences, alongside their corresponding language descriptions. Extensive experiments conducted in the simulation and on the real robots demonstrate that PhyGrasp achieves state-of-the-art performance, particularly in long-tailed cases, e.g., about 10% improvement in success rate over GraspNet. Project page: https://sites.google.com/view/phygrasp
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
