arXiv 2507.218882025-12-11cs.CV

CAPE：一种基于CLIP的互补热图线索点集用于具身参照理解

CAPE: A CLIP-Aware Pointing Ensemble of Complementary Heatmap Cues for Embodied Reference Understanding

Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Istanbul Technical University（伊斯坦布尔技术大学）；Carnegie Mellon University（卡内基梅隆大学）；KIT Campus Transfer GmbH (KCT)（KIT校园转移有限责任公司）

Fevziye Irem Eyiokur, Dogucan Yaman, Hazım Kemal Ekenel, Alexander Waibel

备注：: Accepted by WACV 2026

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

CAPE通过双模型框架和CLIP-aware Pointing Ensemble模块，提升具身参照理解任务中指向线索的多模态推理能力，实现75.0 mAP的高精度表现。

摘要：

我们针对具身参照理解任务进行研究，该任务是通过指向手势和语言预测场景中人物所指的对象。这需要对文本、视觉指向线索和场景上下文进行多模态推理，但现有方法往往未能充分利用视觉消歧信号。我们还观察到，虽然参照物通常与头部到指尖方向一致，但在许多情况下它更接近手腕到指尖方向，使单线假设过于限制。为此，我们提出了一种双模型框架，其中一个是从头部到指尖方向学习，另一个是从手腕到指尖方向学习。我们引入了一种高斯射线热图表示这些线条，并将其作为输入提供强监督信号，以促使模型更好地关注指向线索。为了融合其互补优势，我们提出了CLIP-aware Pointing Ensemble模块，该模块通过CLIP特征进行混合融合。我们进一步加入了一个辅助对象中心预测头以增强参照物定位。我们在YouRefIt上验证了我们的方法，取得了75.0 mAP在0.25 IoU下的表现，同时在CLIP和C_D分数上达到最先进的水平，并在未见过的CAESAR和ISL Pointing上展示了其通用性，证明了其在多个基准上的鲁棒性能。

英文摘要：

We address Embodied Reference Understanding, the task of predicting the object a person in the scene refers to through pointing gesture and language. This requires multimodal reasoning over text, visual pointing cues, and scene context, yet existing methods often fail to fully exploit visual disambiguation signals. We also observe that while the referent often aligns with the head-to-fingertip direction, in many cases it aligns more closely with the wrist-to-fingertip direction, making a single-line assumption overly limiting. To address this, we propose a dual-model framework, where one model learns from the head-to-fingertip direction and the other from the wrist-to-fingertip direction. We introduce a Gaussian ray heatmap representation of these lines and use them as input to provide a strong supervisory signal that encourages the model to better attend to pointing cues. To fuse their complementary strengths, we present the CLIP-Aware Pointing Ensemble module, which performs a hybrid ensemble guided by CLIP features. We further incorporate an auxiliary object center prediction head to enhance referent localization. We validate our approach on YouRefIt, achieving 75.0 mAP at 0.25 IoU, alongside state-of-the-art CLIP and C_D scores, and demonstrate its generality on unseen CAESAR and ISL Pointing, showing robust performance across benchmarks.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。