GLaVE-Cap：基于视觉专家集成的全局-局部对齐视频描述生成
GLaVE-Cap: Global-Local Aligned Video Captioning with Vision Expert Integration
AI总结：
针对视频描述局部到全局范式细节不足且上下文不一致的问题，提出全局-局部对齐框架GLaVE-Cap，通过TrackFusion和CaptionBridge模块结合视觉专家生成连贯细粒度描述，并构建了GLaVE-Bench基准与GLaVE-1.2M数据集，在四个基准上达到SOTA。
摘要：
视频详细描述生成旨在生成全面的视频描述以促进视频理解。近期，视频详细描述生成社区的大部分研究致力于局部到全局的范式，即先从视频片段生成局部描述，再将其总结为全局描述。然而，我们发现该范式会导致描述细节减少且上下文不一致，这可归因于：（1）缺乏确保细粒度描述的机制；（2）局部与全局描述间的交互较弱。为解决上述两个问题，我们提出了GLaVE-Cap，一个集成了视觉专家的全局-局部对齐描述生成框架，它包含两个核心模块：TrackFusion利用视觉专家获取跨帧视觉提示，并结合双流结构，实现全面的局部描述生成；而CaptionBridge利用全局上下文指导局部描述生成，并自适应地将局部描述总结为连贯的全局描述。此外，我们构建了GLaVE-Bench，一个全面的视频描述生成基准，其每个视频的查询数量比现有基准多5倍，覆盖多种视觉维度以促进可靠评估。我们还提供了训练数据集GLaVE-1.2M，包含1.6万条高质量细粒度视频描述和120万个相关问答对。在四个基准上的大量实验表明，我们的GLaVE-Cap取得了最先进的性能。此外，消融研究和学生模型分析进一步验证了所提模块的有效性以及GLaVE-1.2M对视频理解社区的贡献。源代码、模型权重、基准和数据集将开源。
英文摘要：
Video detailed captioning aims to generate comprehensive video descriptions to facilitate video understanding. Recently, most efforts in the video detailed captioning community have been made towards a local-to-global paradigm, which first generates local captions from video clips and then summarizes them into a global caption. However, we find this paradigm leads to less detailed and contextual-inconsistent captions, which can be attributed to (1) no mechanism to ensure fine-grained captions, and (2) weak interaction between local and global captions. To remedy the above two issues, we propose GLaVE-Cap, a Global-Local aligned framework with Vision Expert integration for Captioning, which consists of two core modules: TrackFusion enables comprehensive local caption generation, by leveraging vision experts to acquire cross-frame visual prompts, coupled with a dual-stream structure; while CaptionBridge establishes a local-global interaction, by using global context to guide local captioning, and adaptively summarizing local captions into a coherent global caption. Besides, we construct GLaVE-Bench, a comprehensive video captioning benchmark featuring 5X more queries per video than existing benchmarks, covering diverse visual dimensions to facilitate reliable evaluation. We further provide a training dataset GLaVE-1.2M containing 16K high-quality fine-grained video captions and 1.2M related question-answer pairs. Extensive experiments on four benchmarks show that our GLaVE-Cap achieves state-of-the-art performance. Besides, the ablation studies and student model analyses further validate the effectiveness of the proposed modules and the contribution of GLaVE-1.2M to the video understanding community. The source code, model weights, benchmark, and dataset will be open-sourced.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
