LFS: 用于事件感知和时间多样化的视频描述生成的可学习帧选择器
LFS: Learnable Frame Selector for Event-Aware and Temporally Diverse Video Captioning
AI总结：
本文提出LFS，通过学习选择时间多样且事件相关的帧，提升视频描述的细节质量，在两个基准和ICH-CC上取得显著提升。
摘要：
视频描述生成模型将帧转换为视觉标记，并利用大语言模型（LLMs）生成描述。由于编码所有帧成本过高，通常采用均匀采样，但此方法强制等时间覆盖而忽略事件分布的不均匀性。为此，本文提出可学习帧选择器（LFS），选择时间多样且事件相关的帧。LFS显式建模时间重要性以平衡时间多样性和事件相关性，并采用分层策略确保时间覆盖同时避免聚类。关键在于LFS利用冻结视频-LLMs的描述反馈来学习帧选择，直接优化下游描述质量。此外，本文识别现有基准与人类认知之间的差距，因此引入ICH-CC，由注释者精心设计的问题反映人类一致的视频理解。实验表明，LFS在两个代表性社区基准和ICH-CC上一致提升详细视频描述质量，在VDC上提升2.0%，在ICH-CC上提升超过4%。此外，观察到LFS增强的描述在视频问答中表现更优。总体而言，LFS为详细视频描述生成提供了一个有效且易于集成的解决方案。
英文摘要：
Video captioning models convert frames into visual tokens and generate descriptions with large language models (LLMs). Since encoding all frames is prohibitively expensive, uniform sampling is the default choice, but it enforces equal temporal coverage while ignoring the uneven events distribution. This motivates a Learnable Frame Selector (LFS) that selects temporally diverse and event-relevant frames. LFS explicitly models temporal importance to balance temporal diversity and event relevance, and employs a stratified strategy to ensure temporal coverage while avoiding clustering. Crucially, LFS leverages caption feedback from frozen video-LLMs to learn frame selection that directly optimizes downstream caption quality. Additionally, we identify the gap between existing benchmark and human's cognition. Thus, we introduce ICH-CC built from carefully designed questions by annotators that reflect human-consistent understanding of video. Experiments indicate that LFS consistently improves detailed video captioning across two representative community benchmarks and ICH-CC, achieving up to 2.0% gains on VDC and over 4% gains on ICH-CC. Moreover, we observe that enhanced captions with LFS leads to improved performance on video question answering. Overall, LFS provides an effective and easy-to-integrate solution for detailed video captioning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
