VideoAfford: 通过多模态大语言模型实现人类-物体交互视频中的3D affordance grounding
VideoAfford: Grounding 3D Affordance from Human-Object-Interaction Videos via Multimodal Large Language Model
AI总结：
VideoAfford通过多模态大语言模型实现人类-物体交互视频中的3D affordance grounding，结合动态交互先验和空间感知损失函数，提升机器人操作的可操作区域识别能力。
摘要：
3D affordance grounding旨在突出3D物体上的可操作区域，这对于机器人操作至关重要。先前的研究主要集中在从静态线索如语言和图像中学习affordance知识，这些方法难以提供足够的动态交互上下文来揭示时间性和因果性线索。为缓解这一困境，我们收集了一个全面的基于视频的3D affordance数据集，称为VIDA，其中包含38K个人-物体交互视频，覆盖16种affordance类型、38种物体类别和22K点云。基于VIDA，我们提出了一个强大的基线：VideoAfford，它通过额外的affordance分割能力激活多模态大语言模型，使模型能够在统一框架中实现世界知识推理和细粒度affordance grounding。为增强动作理解能力，我们利用潜在动作编码器从HOI视频中提取动态交互先验。此外，我们引入了一种空间感知的损失函数，使VideoAfford能够获得全面的3D空间知识。广泛的实验评估表明，我们的模型显著优于已建立的方法，并在affordance推理能力方面表现出强大的开放世界泛化能力。所有数据集和代码将被公开发布，以促进该领域研究的进一步发展。
英文摘要：
3D affordance grounding aims to highlight the actionable regions on 3D objects, which is crucial for robotic manipulation. Previous research primarily focused on learning affordance knowledge from static cues such as language and images, which struggle to provide sufficient dynamic interaction context that can reveal temporal and causal cues. To alleviate this predicament, we collect a comprehensive video-based 3D affordance dataset, \textit{VIDA}, which contains 38K human-object-interaction videos covering 16 affordance types, 38 object categories, and 22K point clouds. Based on \textit{VIDA}, we propose a strong baseline: VideoAfford, which activates multimodal large language models with additional affordance segmentation capabilities, enabling both world knowledge reasoning and fine-grained affordance grounding within a unified framework. To enhance action understanding capability, we leverage a latent action encoder to extract dynamic interaction priors from HOI videos. Moreover, we introduce a \textit{spatial-aware} loss function to enable VideoAfford to obtain comprehensive 3D spatial knowledge. Extensive experimental evaluations demonstrate that our model significantly outperforms well-established methods and exhibits strong open-world generalization with affordance reasoning abilities. All datasets and code will be publicly released to advance research in this area.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
