arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2411.08451cs.CV
收藏

AD-DINO:用于距离感知具身指代理解的注意力动态 DINO

AD-DINO: Attention-Dynamic DINO for Distance-Aware Embodied Reference Understanding

Harbin Institute of Technology(哈尔滨工业大学);Nanjing University of Information Science & Technology(南京信息工程大学)

Hao Guo, Wei Fan, Baichun Wei, Jianfei Zhu, Jin Tian, Chunzhi Yi, Feng Jiang

AI总结:

本文提出 Attention-Dynamic DINO 框架,通过结合视觉文本特征预测目标边界框与注意力源,并引入注意力动态触摸线机制,在 YouRefIt 数据集上显著提升了具身指代理解性能并超越人类表现。

摘要:

具身指代理解对于智能体通过手势信号和语言描述基于人类意图预测指代物至关重要。本文介绍了 Attention-Dynamic DINO,这是一个旨在缓解各种交互情境下指示手势误解的新颖框架。我们的方法整合视觉和文本特征,以同时预测目标对象的边界框和指示手势中的注意力源。利用视觉视角采集中非语言交流的距离感知特性,我们扩展了虚拟触摸线机制,并提出了一种注意力动态触摸线,以基于交互距离表示指示手势。这种距离感知方法与注意力源的独立预测相结合,增强了对象与手势表示线之间的对齐。在 YouRefIt 数据集上的大量实验证明了我们的手势信息理解方法在显著提升任务性能方面的有效性。我们的模型在 0.25 IoU 阈值下实现了 76.4% 的准确率,并在 0.75 IoU 阈值下显著超越了人类表现,标志着该领域的首次突破。与以往研究中距离不感知理解方法的对比实验,进一步验证了 Attention-Dynamic Touch Line 在多样化情境下的优越性。

英文摘要:

Embodied reference understanding is crucial for intelligent agents to predict referents based on human intention through gesture signals and language descriptions. This paper introduces the Attention-Dynamic DINO, a novel framework designed to mitigate misinterpretations of pointing gestures across various interaction contexts. Our approach integrates visual and textual features to simultaneously predict the target object's bounding box and the attention source in pointing gestures. Leveraging the distance-aware nature of nonverbal communication in visual perspective taking, we extend the virtual touch line mechanism and propose an attention-dynamic touch line to represent referring gesture based on interactive distances. The combination of this distance-aware approach and independent prediction of the attention source, enhances the alignment between objects and the gesture represented line. Extensive experiments on the YouRefIt dataset demonstrate the efficacy of our gesture information understanding method in significantly improving task performance. Our model achieves 76.4% accuracy at the 0.25 IoU threshold and, notably, surpasses human performance at the 0.75 IoU threshold, marking a first in this domain. Comparative experiments with distance-unaware understanding methods from previous research further validate the superiority of the Attention-Dynamic Touch Line across diverse contexts.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑