HAMMER: 通过跨模态整合利用大语言模型进行意图驱动的3D affordance grounding
HAMMER: Harnessing MLLM via Cross-Modal Integration for Intention-Driven 3D Affordance Grounding
AI总结：
HAMMER通过跨模态整合多模态大语言模型，实现意图驱动的3D affordance grounding，提升3D表示的准确性和鲁棒性。
摘要：
人类通常通过图像或视频中观察到的交互来识别3D物体的 affordance，并且一旦形成，这种知识可以泛化到新物体。受此原理启发，我们提出了一种新的框架，利用新兴的多模态大语言模型（MLLMs）进行交互意图驱动的3D affordance grounding，即HAMMER。不同于生成显式的物体属性描述或依赖现成的2D分割器，我们替代地将图像中描述的交互意图聚合为一种接触感知的嵌入，并引导模型推断文本 affordance 标签，确保其彻底挖掘物体语义和上下文线索。我们进一步设计了一种层次化的跨模态整合机制，以充分利用MLLM中的互补信息进行3D表示细化，并引入一个多粒度的几何提升模块，将空间特征注入到提取的意图嵌入中，从而促进准确的3D affordance 定位。在公共数据集和我们新构建的损坏基准上的广泛实验表明，HAMMER相比现有方法具有优越性和鲁棒性。所有代码和权重均已公开。
英文摘要：
Humans commonly identify 3D object affordance through observed interactions in images or videos, and once formed, such knowledge can be generically generalized to novel objects. Inspired by this principle, we advocate for a novel framework that leverages emerging multimodal large language models (MLLMs) for interaction intention-driven 3D affordance grounding, namely HAMMER. Instead of generating explicit object attribute descriptions or relying on off-the-shelf 2D segmenters, we alternatively aggregate the interaction intention depicted in the image into a contact-aware embedding and guide the model to infer textual affordance labels, ensuring it thoroughly excavates object semantics and contextual cues. We further devise a hierarchical cross-modal integration mechanism to fully exploit the complementary information from the MLLM for 3D representation refinement and introduce a multi-granular geometry lifting module that infuses spatial characteristics into the extracted intention embedding, thus facilitating accurate 3D affordance localization. Extensive experiments on public datasets and our newly constructed corrupted benchmark demonstrate the superiority and robustness of HAMMER compared to existing approaches. All code and weights are publicly available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
