使用大规模预训练模型的免训练视频时序定位
Training-free Video Temporal Grounding using Large-scale Pre-trained Models
AI总结：
提出免训练视频时序定位方法TFVTG，利用LLMs分析查询文本中子事件的时序关系，结合VLMs的动态与静态评分函数评估相关性并定位整合候选片段，在零样本及跨数据集设置下实现最优性能。
摘要：
视频时序定位旨在从未剪辑视频中识别出与给定自然语言查询最相关的视频片段。现有的视频时序定位模型依赖特定数据集进行训练，数据收集成本高，且在跨数据集和分布外（OOD）设置下泛化能力差。本文提出一种免训练视频时序定位（TFVTG）方法，利用预训练大模型的能力。朴素基线方法是在视频中枚举候选片段，并使用预训练视觉语言模型（VLMs）根据视觉-语言对齐选择最佳候选。然而，现有大多数VLMs基于图像-文本对或已剪辑视频片段-文本对训练，导致其难以（1）理解同一视频内多个事件的关系并区分其时序边界；（2）理解并对视频中事件的动态转换（从一个事件到另一个事件的转换）保持敏感。为解决这些问题，我们利用大语言模型（LLMs）分析查询文本中包含的多个子事件，并分析这些事件的时序顺序和关系。其次，我们将子事件拆分为动态转换和静态状态部分，提出使用VLMs的动态和静态评分函数，以更好地评估事件与描述之间的相关性。最后，针对每个子事件描述，我们使用VLMs定位top-k候选片段，并利用LLMs提供的子事件顺序和关系来过滤和整合这些候选片段。我们的方法在Charades-STA和ActivityNet Captions数据集上的零样本视频时序定位任务中取得最佳性能，无需任何训练，并在跨数据集和OOD设置中展现出更优的泛化能力。
英文摘要：
Video temporal grounding aims to identify video segments within untrimmed videos that are most relevant to a given natural language query. Existing video temporal localization models rely on specific datasets for training and have high data collection costs, but they exhibit poor generalization capability under the across-dataset and out-of-distribution (OOD) settings. In this paper, we propose a Training-Free Video Temporal Grounding (TFVTG) approach that leverages the ability of pre-trained large models. A naive baseline is to enumerate proposals in the video and use the pre-trained visual language models (VLMs) to select the best proposal according to the vision-language alignment. However, most existing VLMs are trained on image-text pairs or trimmed video clip-text pairs, making it struggle to (1) grasp the relationship and distinguish the temporal boundaries of multiple events within the same video; (2) comprehend and be sensitive to the dynamic transition of events (the transition from one event to another) in the video. To address these issues, we propose leveraging large language models (LLMs) to analyze multiple sub-events contained in the query text and analyze the temporal order and relationships between these events. Secondly, we split a sub-event into dynamic transition and static status parts and propose the dynamic and static scoring functions using VLMs to better evaluate the relevance between the event and the description. Finally, for each sub-event description, we use VLMs to locate the top-k proposals and leverage the order and relationships between sub-events provided by LLMs to filter and integrate these proposals. Our method achieves the best performance on zero-shot video temporal grounding on Charades-STA and ActivityNet Captions datasets without any training and demonstrates better generalization capabilities in cross-dataset and OOD settings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
