arXivDaily arXiv每日学术速递 周一至周五更新
​通知:联邦节假日调休:哥伦布日 / 原住民日(Columbus Day / Indigenous Peoples' Day),今日(13日)无更新。
arXiv 2603.17647cs.CV

面向部分的开放词汇3D功能接地 via 语义和几何对齐

Part-Aware Open-Vocabulary 3D Affordance Grounding via Prototypical Semantic and Geometric Alignment

ShanghaiTech University(上海科技大学)

Dongqiang Gou, Xuming He

AI总结:

本文提出一种双阶段跨模态框架,通过增强语义和几何表示,解决开放词汇3D功能接地中的语义一致性、几何对齐和开放词汇泛化问题。

摘要:

将自然语言问题接地到3D物体功能相关区域--称为语言驱动的3D功能接地--对于具身智能和人机交互至关重要。现有方法虽然从基于标签的方法发展到语言驱动方法,但仍面临开放词汇泛化、细粒度几何对齐和部分级语义一致性方面的挑战。为了解决这些问题,我们提出了一种新的双阶段跨模态框架,以增强开放词汇3D功能接地的语义和几何表示。在第一阶段,大型语言模型生成具有部分意识的指令以恢复缺失的语义,使模型能够链接语义相似的功能。在第二阶段,我们引入了两个关键组件:功能原型聚合(APA),它捕捉每个功能的跨物体几何一致性,以及物体内关系建模(IORM),它细化物体内的几何区分以支持精确的语义对齐。我们通过在新引入的基准以及两个现有基准上的广泛实验验证了我们方法的有效性,证明其在与现有方法相比具有优越性能。

英文摘要:

Grounding natural language questions to functionally relevant regions in 3D objects -- termed language-driven 3D affordance grounding -- is essential for embodied intelligence and human-AI interaction. Existing methods, while progressing from label-based to language-driven approaches, still face challenges in open-vocabulary generalization, fine-grained geometric alignment, and part-level semantic consistency. To address these issues, we propose a novel two-stage cross-modal framework that enhances both semantic and geometric representations for open-vocabulary 3D affordance grounding. In the first stage, large language models generate part-aware instructions to recover missing semantics, enabling the model to link semantically similar affordances. In the second stage, we introduce two key components: Affordance Prototype Aggregation (APA), which captures cross-object geometric consistency for each affordance, and Intra-Object Relational Modeling (IORM), which refines geometric differentiation within objects to support precise semantic alignment. We validate the effectiveness of our method through extensive experiments on a newly introduced benchmark, as well as two existing benchmarks, demonstrating superior performance in comparison with existing methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑