arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.09511cs.CV
收藏

GEAL:基于跨模态一致性的可泛化3D可供性学习

GEAL: Generalizable 3D Affordance Learning with Cross-Modal Consistency

National University of Singapore(新加坡国立大学)

Dongyue Lu, Lingdong Kong, Tianxin Huang, Gim Hee Lee

AI总结:

针对现有3D可供性学习泛化性与鲁棒性不足的问题,提出GEAL框架,通过双分支架构与跨模态对齐模块迁移2D预训练模型语义知识,还构建新基准验证其在多场景下的性能优势。

摘要:

从语义线索中识别3D物体上的可供性区域,对机器人技术与人机交互至关重要。然而,现有3D可供性学习方法因标注数据有限,且依赖侧重几何编码的3D骨干网络,往往对真实世界噪声和数据损坏缺乏抵御能力,在泛化性与鲁棒性上存在局限。我们提出GEAL这一新型框架,旨在利用大规模预训练2D模型提升3D可供性学习的泛化性与鲁棒性。我们采用带高斯溅射的双分支架构,建立3D点云与2D表征之间的一致映射,实现从稀疏点云生成逼真的2D渲染图。粒度自适应融合模块与2D-3D一致性对齐模块进一步强化跨模态对齐与知识迁移,使3D分支能够受益于2D模型丰富的语义信息与泛化能力。为全面评估鲁棒性,我们提出两个基于数据损坏的新基准:PIAD-C与LASO-C。在公开数据集及我们提出的基准上开展的大量实验表明,GEAL在已见过和新的物体类别上,以及损坏数据上,性能始终优于现有方法,证明其在多样条件下能实现鲁棒且适配性强的可供性预测。代码与损坏数据集已公开发布。

英文摘要:

Identifying affordance regions on 3D objects from semantic cues is essential for robotics and human-machine interaction. However, existing 3D affordance learning methods struggle with generalization and robustness due to limited annotated data and a reliance on 3D backbones focused on geometric encoding, which often lack resilience to real-world noise and data corruption. We propose GEAL, a novel framework designed to enhance the generalization and robustness of 3D affordance learning by leveraging large-scale pre-trained 2D models. We employ a dual-branch architecture with Gaussian splatting to establish consistent mappings between 3D point clouds and 2D representations, enabling realistic 2D renderings from sparse point clouds. A granularity-adaptive fusion module and a 2D-3D consistency alignment module further strengthen cross-modal alignment and knowledge transfer, allowing the 3D branch to benefit from the rich semantics and generalization capacity of 2D models. To holistically assess the robustness, we introduce two new corruption-based benchmarks: PIAD-C and LASO-C. Extensive experiments on public datasets and our benchmarks show that GEAL consistently outperforms existing methods across seen and novel object categories, as well as corrupted data, demonstrating robust and adaptable affordance prediction under diverse conditions. Code and corruption datasets have been made publicly available.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑