TA-Prompting: 通过时间锚点增强视频大语言模型以实现密集视频描述
TA-Prompting: Enhancing Video Large Language Models for Dense Video Captioning via Temporal Anchors
AI总结：
TA-Prompting通过引入时间锚点提升视频大语言模型的密集视频描述能力，有效解决时间边界识别问题，提升时间感知视频事件理解性能。
摘要：
密集视频描述旨在对输入视频中所有时间局部化事件进行解释和描述。近期最先进的方法利用大语言模型（LLMs）为视频数据提供详细的时刻描述。然而，现有视频大语言模型在识别未剪辑视频中的精确事件边界方面仍然具有挑战性，导致生成的描述不准确。在本文中，我们提出了TA-Prompting，通过时间锚点增强视频大语言模型，使模型能够精确定位事件并提示视频大语言模型进行时间感知的视频事件理解。在推理过程中，为了正确确定从视频中任意数量事件生成的输出描述序列，我们引入了一种事件连贯采样策略，以选择具有足够连贯性的时间事件和跨模态相似性的事件描述。通过在基准数据集上的广泛实验，我们证明了我们的TA-Prompting在对抗最先进的视频大语言模型方面具有优势，在密集视频描述和时间理解任务中，包括时刻检索和时间问答任务，均取得了优越的性能。
英文摘要：
Dense video captioning aims to interpret and describe all temporally localized events throughout an input video. Recent state-of-the-art methods leverage large language models (LLMs) to provide detailed moment descriptions for video data. However, existing VideoLLMs remain challenging in identifying precise event boundaries in untrimmed videos, causing the generated captions to be not properly grounded. In this paper, we propose TA-Prompting, which enhances VideoLLMs via Temporal Anchors that learn to precisely localize events and prompt the VideoLLMs to perform temporal-aware video event understanding. During inference, in order to properly determine the output caption sequence from an arbitrary number of events presented within a video, we introduce an event coherent sampling strategy to select event captions with sufficient coherence across temporal events and cross-modal similarity with the given video. Through extensive experiments on benchmark datasets, we show that our TA-Prompting is favorable against state-of-the-art VideoLLMs, yielding superior performance on dense video captioning and temporal understanding tasks including moment retrieval and temporalQA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
