LongEgoRefer: 长形式自我中心视频指代表达理解基准
LongEgoRefer: A Benchmark for Long-Form Egocentric Video Referring Expression Comprehension
AI总结：
提出LongEgoRefer基准，基于Ego4D长视频构建，解决现有基准仅关注短视频导致目标稀疏和活动复杂的问题，评估现有方法发现性能显著下降，凸显长形式自我中心时空定位的挑战。
摘要：
自我中心视频捕捉了丰富多样的人-物交互，已成为理解与物体相关的人类活动的基础资源。在此背景下，视频指代表达理解（Video REC）任务——根据自然语言查询在视频帧中定位所指物体的时间和空间范围——在将文本描述与未修剪的自我中心记录中的观察物体联系起来方面发挥着关键作用。然而，现有的自我中心视频REC基准主要关注短视频片段，其中某些目标物体在帧中密集出现。这种设置不能反映现实世界的自我中心记录，后者是长形式、未修剪的，并以稀疏的物体出现和复杂的活动转换为特征。为解决这一局限性，我们引入了LongEgoRefer，这是一个基于Ego4D数据集中长形式视频构建的新颖且具有挑战性的基准。LongEgoRefer包含1,498个指代表达，平均视频时长为45分钟。该基准表现出极端的物体稀疏性、详细的语言描述以及嵌入在长动态自我中心叙事中的复杂人-物交互。因此，它定义了一个要求模型同时识别事件发生时间和所指物体在扩展视频序列中出现位置的苛刻时空定位问题。我们评估了现有的Video REC方法，包括基于视觉语言模型与Grounded SAM2结合的无训练基线。大量实验表明，即使先进的基线和当前最先进的模型在LongEgoRefer上也表现挣扎。这些结果凸显了长形式自我中心时空定位的内在难度，并强调了需要更鲁棒的视频理解模型。
英文摘要：
Egocentric videos capture rich and diverse human-object interactions and have emerged as a fundamental resource for understanding human activities related to objects. In this context, Video Referring Expression Comprehension (Video REC), the task of localizing the temporal and spatial extent of a referred object in video frames given a natural language query, plays a key role in linking textual descriptions to observed objects in untrimmed egocentric recordings. However, existing egocentric Video REC benchmarks primarily focus on short video clips, where some target object appears densely within frames. Such settings do not reflect real-world egocentric recordings, which are long-form, untrimmed, and characterized by sparse object occurrences and complex activity transitions. To address this limitation, we introduce LongEgoRefer, a novel and challenging benchmark constructed from long-form videos in the Ego4D dataset. LongEgoRefer contains 1,498 referring expressions with an average video duration of 45 minutes. The benchmark exhibits extreme target sparsity, detailed linguistic descriptions, and complex human-object interactions embedded in long, dynamic egocentric narratives. Consequently, it defines a demanding spatio-temporal grounding problem that requires models to identify both when an event occurs and where the referred object appears within extended video sequences. We evaluate existing Video REC approaches, including training-free baselines based on vision-language models combined with Grounded SAM2. Extensive experiments show that even advanced baselines and current state-of-the-art models struggle significantly on LongEgoRefer. These results highlight the intrinsic difficulty of long-form egocentric spatio-temporal grounding and emphasize the need for more robust video understanding models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
