arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2210.05668cs.CV
收藏

用接触线式Transformer理解接地基准

Understanding Embodied Reference with Touch-Line Transformer

Institute for AI Industry Research, Tsinghua University(清华大学智能产业研究院);UCSD(加州大学圣地亚哥分校);Intel Labs China(英特尔中国研究院);Peking University(北京大学);Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院)

Yang Li, Xiaoxue Chen, Hao Zhao, Jiangtao Gong, Guyue Zhou, Federico Rossano, Yixin Zhu

摘要:

我们研究具体的参考理解,定位使用具体的手势信号和语言参考的任务。人类研究表明,所指或指向的物体并不位于肘腕线上,这是一种常见的误解;相反,它们位于所谓的虚拟触摸线上。然而,现有的人体姿势表示未能结合虚拟触摸线。为了解决这个问题,我们设计了接触线Transformer:它作为输入标记化的视觉和文本特征,同时预测所指对象的限界框和接触线向量。利用这种接触线先验,我们进一步设计了一个几何一致性损失,鼓励参考和接触线之间的共线性。使用触摸线作为手势信息显著提高了模型性能。在YouRefIt数据集上的实验表明,我们的方法在0.75 IoU标准下实现了+25.0%的准确率提高,缩小了模型和人类性能之间的63.6%差距。此外,我们计算验证以前的人类研究表明,计算模型更准确地定位参考时,使用虚拟触摸线比使用肘腕线。

英文摘要:

We study embodied reference understanding, the task of locating referents using embodied gestural signals and language references. Human studies have revealed that objects referred to or pointed to do not lie on the elbow-wrist line, a common misconception; instead, they lie on the so-called virtual touch line. However, existing human pose representations fail to incorporate the virtual touch line. To tackle this problem, we devise the touch-line transformer: It takes as input tokenized visual and textual features and simultaneously predicts the referent's bounding box and a touch-line vector. Leveraging this touch-line prior, we further devise a geometric consistency loss that encourages the co-linearity between referents and touch lines. Using the touch-line as gestural information improves model performances significantly. Experiments on the YouRefIt dataset show our method achieves a +25.0% accuracy improvement under the 0.75 IoU criterion, closing 63.6% of the gap between model and human performances. Furthermore, we computationally verify prior human studies by showing that computational models more accurately locate referents when using the virtual touch line than when using the elbow-wrist line.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑