arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.17488cs.ROcs.CVcs.LG
收藏

GenDP:面向类别级可泛化扩散策略的3D语义场

GenDP: 3D Semantic Fields for Category-Level Generalizable Diffusion Policy

Columbia University(哥伦比亚大学);University of Illinois, Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Boston Dynamics AI Institute(波士顿动力人工智能研究所)

Yixuan Wang, Guang Yin, Binghao Huang, Tarik Kelestemur, Jiuguang Wang, Yunzhu Li

AI总结:

GenDP通过3D语义场为扩散策略引入显式几何与语义信息,利用基础视觉模型从多视角RGBD生成描述子场并与参考比较得到语义场,在八项类别级操作任务上将未见实例平均成功率从20%提升至93%。

摘要:

基于扩散的策略在执行复杂机器人操作任务方面展现出卓越能力,但缺乏对几何与语义的显式表征,这通常限制了其泛化到未见物体和布局的能力。为增强扩散策略(Diffusion Policy)的泛化能力,我们提出一种新颖框架,通过3D语义场引入显式的空间与语义信息。我们利用大型基础视觉模型从多视角RGBD观测中生成3D描述子场,然后将这些描述子场与参考描述子进行比较以获得语义场。所提方法显式考虑几何与语义,从而在需要类别级泛化、解决几何歧义以及关注细微几何细节的任务中具备强大的泛化能力。我们在涉及多个物体类别的铰接物体以及具有不同形状和纹理的实例的八项任务上评估了我们的方法。我们的方法将扩散策略在未见实例上的平均成功率从20%提高到93%,证明了其有效性。此外,我们提供了详细的分析和可视化,以解释性能提升的来源,并说明我们的方法如何泛化到新实例。

英文摘要:

Diffusion-based policies have shown remarkable capability in executing complex robotic manipulation tasks but lack explicit characterization of geometry and semantics, which often limits their ability to generalize to unseen objects and layouts. To enhance the generalization capabilities of Diffusion Policy, we introduce a novel framework that incorporates explicit spatial and semantic information via 3D semantic fields. We generate 3D descriptor fields from multi-view RGBD observations with large foundational vision models, then compare these descriptor fields against reference descriptors to obtain semantic fields. The proposed method explicitly considers geometry and semantics, enabling strong generalization capabilities in tasks requiring category-level generalization, resolving geometric ambiguities, and attention to subtle geometric details. We evaluate our method across eight tasks involving articulated objects and instances with varying shapes and textures from multiple object categories. Our method demonstrates its effectiveness by increasing Diffusion Policy's average success rate on unseen instances from 20% to 93%. Additionally, we provide a detailed analysis and visualization to interpret the sources of performance gain and explain how our method can generalize to novel instances.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑