VideoDetective：通过外在查询和内在相关性进行长视频理解的线索搜索
VideoDetective: Clue Hunting via both Extrinsic Query and Intrinsic Relevance for Long Video Understanding
AI总结：
本文提出VideoDetective框架，通过结合查询与片段的相关性及片段间亲和力，有效定位长视频问答中的关键片段，提升多模态大语言模型在长视频理解中的性能。
摘要：
长视频理解对多模态大语言模型（MLLMs）而言仍具挑战性，由于上下文窗口有限，需要识别稀疏的查询相关视频片段。然而，现有方法主要基于查询本地化线索，忽视了视频的内在结构和片段间变化的相关性。为此，我们提出VideoDetective框架，通过整合查询到片段的相关性和片段间亲和力，实现长视频问答中的有效线索搜索。具体而言，我们将视频划分为多个片段，并通过视觉相似性和时间接近性构建视觉-时间亲和图。然后执行假设-验证-细化循环，估算观察片段与查询的相关性得分，并将这些得分传播到未观察片段，从而获得全局相关性分布，指导最终答案中关键片段的定位。实验表明，我们的方法在主流MLLMs上广泛基准测试中均取得显著提升，VideoMME-long数据集上的准确率提升高达7.5%。代码可在https://videodetective.github.io/获取。
英文摘要：
Long video understanding remains challenging for multimodal large language models (MLLMs) due to limited context windows, which necessitate identifying sparse query-relevant video segments. However, existing methods predominantly localize clues based solely on the query, overlooking the video's intrinsic structure and varying relevance across segments. To address this, we propose VideoDetective, a framework that integrates query-to-segment relevance and inter-segment affinity for effective clue hunting in long-video question answering. Specifically, we divide a video into various segments and represent them as a visual-temporal affinity graph built from visual similarity and temporal proximity. We then perform a Hypothesis-Verification-Refinement loop to estimate relevance scores of observed segments to the query and propagate them to unseen segments, yielding a global relevance distribution that guides the localization of the most critical segments for final answering with sparse observation. Experiments show our method consistently achieves substantial gains across a wide range of mainstream MLLMs on representative benchmarks, with accuracy improvements of up to 7.5% on VideoMME-long. Our code is available at https://videodetective.github.io/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
