通过多模态引导在长视频中定位时刻
Localizing Moments in Long Video Via Multimodal Guidance
AI总结：
本文提出一种多模态引导框架，通过修剪不可描述窗口提升长视频中自然语言定位性能，在MAD和Ego4D上分别超越最先进模型4.1%和4.52%。
摘要：
大规模、长格式的MAD和Ego4D数据集的近期引入，使研究人员能够调查当前最先进方法在长格式设置下进行视频定位的性能，并获得了有趣的发现：当前仅靠定位方法无法应对这一挑战性任务和设置，因为它们无法处理长视频序列。在本文中，我们提出了一种通过识别并修剪不可描述窗口来提高长视频中自然语言定位性能的方法。我们设计了一个由引导模型和基础定位模型组成的引导定位框架。引导模型强调可描述窗口，而基础定位模型分析短时间窗口以确定哪些片段准确匹配给定的语言查询。我们为引导模型提供了两种设计：查询无关和查询相关，它们在效率和准确性之间取得平衡。实验表明，我们提出的方法在MAD和Ego4D（NLQ）上分别比最先进模型高出4.1%和4.52%。复现实验所需的代码、数据和MAD的音频特征可在以下网址获取：https://github.com/waybarrios/guidance-based-video-grounding。
英文摘要：
The recent introduction of the large-scale, long-form MAD and Ego4D datasets has enabled researchers to investigate the performance of current state-of-the-art methods for video grounding in the long-form setup, with interesting findings: current grounding methods alone fail at tackling this challenging task and setup due to their inability to process long video sequences. In this paper, we propose a method for improving the performance of natural language grounding in long videos by identifying and pruning out non-describable windows. We design a guided grounding framework consisting of a Guidance Model and a base grounding model. The Guidance Model emphasizes describable windows, while the base grounding model analyzes short temporal windows to determine which segments accurately match a given language query. We offer two designs for the Guidance Model: Query-Agnostic and Query-Dependent, which balance efficiency and accuracy. Experiments demonstrate that our proposed method outperforms state-of-the-art models by 4.1% in MAD and 4.52% in Ego4D (NLQ), respectively. Code, data and MAD's audio features necessary to reproduce our experiments are available at: https://github.com/waybarrios/guidance-based-video-grounding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
