arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2504.09623cs.CVcs.AIcs.MM
收藏

Ges3ViG:将指向手势融入基于语言的3D视觉定位以实现具身参考理解

Ges3ViG: Incorporating Pointing Gestures into Language-Based 3D Visual Grounding for Embodied Reference Understanding

Stony Brook University(石溪大学);BITS Pilani(pilani bits大学);Singapore-MIT Alliance for Research and Technology Centre(新加坡-麻省理工研究与技术联盟中心);IHPC, Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局信息技术研究院);APPCAIR;Massachusetts Institute of Technology(麻省理工学院);Singapore Management University(新加坡管理大学)

Atharv Mahesh Mane, Dulanga Weerakoon, Vigneshwaran Subbaraju, Sougata Sen, Sanjay E. Sarma, Archan Misra

AI总结:

针对3D具身参考理解(3D-ERU)研究有限的问题,提出数据增强框架Imputer构建ImputeRefer基准数据集,并提出模型Ges3ViG,其准确率较其他3D-ERU模型提升约30%,较纯语言模型提升约9%。

摘要:

三维具身参考理解(3D-ERU)结合语言描述和伴随的指向手势,识别3D场景中最相关的目标物体。尽管先前的工作探索了纯基于语言的3D定位,但对同时融入人类指向手势的3D-ERU的研究有限。为解决这一空白,我们引入数据增强框架Imputer,并通过将人类指向手势融入仅含语言指令的现有3D场景数据集,用其构建新的3D-ERU基准数据集ImputeRefer。我们还提出新型3D-ERU模型Ges3ViG,其准确率较其他3D-ERU模型提升约30%,较其他纯语言3D定位模型提升约9%。我们的代码和数据集可在https://github.com/AtharvMane/Ges3ViG获取。

英文摘要:

3-Dimensional Embodied Reference Understanding (3D-ERU) combines a language description and an accompanying pointing gesture to identify the most relevant target object in a 3D scene. Although prior work has explored pure language-based 3D grounding, there has been limited exploration of 3D-ERU, which also incorporates human pointing gestures. To address this gap, we introduce a data augmentation framework-Imputer, and use it to curate a new benchmark dataset-ImputeRefer for 3D-ERU, by incorporating human pointing gestures into existing 3D scene datasets that only contain language instructions. We also propose Ges3ViG, a novel model for 3D-ERU that achieves ~30% improvement in accuracy as compared to other 3D-ERU models and ~9% compared to other purely language-based 3D grounding models. Our code and dataset are available at https://github.com/AtharvMane/Ges3ViG.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑