长时第一人称视频中的基于定位的问答
Grounded Question-Answering in Long Egocentric Videos
AI总结：
本文针对长时第一人称视频中的开放式问答任务，提出统一查询定位与回答的模型，利用大语言模型合成数据，并引入封闭式问答评估，在QaEgo4D和Ego4D-NLQ上取得最先进性能。
摘要：
现有的视频理解方法主要针对第三人称视角的短视频设计，在机器人等特定领域的适用性受到限制。本文深入研究了长时第一人称视频中的开放式问答（QA），使个人或机器人能够查询自身过去的视觉经历。该任务面临独特挑战，包括在大量视频内容中对查询进行时间定位的复杂性、精确数据标注的高资源需求，以及由于开放式答案的模糊性而导致的评估固有困难。我们提出的方法通过以下方式应对这些挑战：（i）将查询定位与回答集成到一个统一模型中，以减少误差传播；（ii）采用大语言模型进行高效且可扩展的数据合成；（iii）引入封闭式问答任务用于评估，以管理答案模糊性。大量实验证明了我们方法的有效性，该方法还在QaEgo4D和Ego4D-NLQ基准上取得了最先进的性能。代码、数据和模型可在https://github.com/Becomebright/GroundVQA获取。
英文摘要：
Existing approaches to video understanding, mainly designed for short videos from a third-person perspective, are limited in their applicability in certain fields, such as robotics. In this paper, we delve into open-ended question-answering (QA) in long, egocentric videos, which allows individuals or robots to inquire about their own past visual experiences. This task presents unique challenges, including the complexity of temporally grounding queries within extensive video content, the high resource demands for precise data annotation, and the inherent difficulty of evaluating open-ended answers due to their ambiguous nature. Our proposed approach tackles these challenges by (i) integrating query grounding and answering within a unified model to reduce error propagation; (ii) employing large language models for efficient and scalable data synthesis; and (iii) introducing a close-ended QA task for evaluation, to manage answer ambiguity. Extensive experiments demonstrate the effectiveness of our method, which also achieves state-of-the-art performance on the QaEgo4D and Ego4D-NLQ benchmarks. Code, data, and models are available at https://github.com/Becomebright/GroundVQA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
