OwlCap：通过HMD-270K与字幕集等价奖励协调视频字幕生成中的运动-细节平衡
OwlCap: Harmonizing Motion-Detail for Video Captioning via HMD-270K and Caption Set Equivalence Reward
AI总结：
针对视频字幕生成中运动与细节不平衡问题，本文提出HMD-270K数据集和字幕集等价奖励CSER，并据此训练出OwlCap多模态大语言模型，在VDC和DREAM-1K基准上显著提升性能。
摘要：
视频字幕生成旨在对视频内容生成全面且连贯的描述，从而推动视频理解与生成的发展。然而，现有方法通常存在运动-细节不平衡的问题，模型往往过度强调其中一个方面而忽视另一方面。这种不平衡会导致字幕不完整，进而造成视频理解与生成缺乏一致性。为解决该问题，我们从两个方面提出方案：1）数据方面：我们通过两阶段流程构建了协调运动-细节270K（HMD-270K）数据集，该流程包括运动-细节融合（MDF）和细粒度检验（FGE）。2）优化方面：我们基于分组相对策略优化（GRPO）引入了字幕集等价奖励（CSER）。CSER通过单元到集合匹配和双向验证，增强了在同时捕捉运动和细节方面的完整性与准确性。基于HMD-270K监督微调以及使用CSER的GRPO后训练，我们开发了OwlCap，这是一个具有运动-细节平衡能力的强大视频字幕多模态大语言模型（MLLM）。实验结果表明，OwlCap在两个基准上相较于基线模型取得了显著改进：注重细节的VDC（+4.2 Acc）和注重运动的DREAM-1K（+4.6 F1）。HMD-270K数据集和OwlCap模型将公开发布，以促进视频字幕研究社区的发展。
英文摘要：
Video captioning aims to generate comprehensive and coherent descriptions of the video content, contributing to the advancement of both video understanding and generation. However, existing methods often suffer from motion-detail imbalance, as models tend to overemphasize one aspect while neglecting the other. This imbalance results in incomplete captions, which in turn leads to a lack of consistency in video understanding and generation. To address this issue, we propose solutions from two aspects: 1) Data aspect: We constructed the Harmonizing Motion-Detail 270K (HMD-270K) dataset through a two-stage pipeline: Motion-Detail Fusion (MDF) and Fine-Grained Examination (FGE). 2) Optimization aspect: We introduce the Caption Set Equivalence Reward (CSER) based on Group Relative Policy Optimization (GRPO). CSER enhances completeness and accuracy in capturing both motion and details through unit-to-set matching and bidirectional validation. Based on the HMD-270K supervised fine-tuning and GRPO post-training with CSER, we developed OwlCap, a powerful video captioning multi-modal large language model (MLLM) with motion-detail balance. Experimental results demonstrate that OwlCap achieves significant improvements compared to baseline models on two benchmarks: the detail-focused VDC (+4.2 Acc) and the motion-focused DREAM-1K (+4.6 F1). The HMD-270K dataset and OwlCap model will be publicly released to facilitate video captioning research community advancements.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
