发表机构
MMLab, CUHK; Peking University（香港中文大学MMLab; 北京大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对音视频联合字幕的数据集、奖励信号、评估基准的局限，提出AVCap-100K数据集、AVCap模型及专用基准指标，通过Da-GRPO优化的AVCap模型性能优异。
AI 中文摘要
详细的音视频联合字幕对于多模态视频理解与生成至关重要。然而，现有工作受限于三大主要局限：（1）缺乏带有细粒度音视频联合字幕的高质量公开数据集；（2）依赖粗粒度奖励信号的强化学习方法；（3）缺乏用于在原子层面评估详细音视频字幕的基准与指标。为应对这些挑战，我们提出：（1）AVCap-100K，一个包含10万条时间对齐、细节丰富的音视频字幕的高质量数据集；（2）AVCap，一个通过细节感知GRPO（Da-GRPO）优化的模型，在开源模型中实现了最优性能，并在多项评估中与专有模型相当或超越；（3）AVCap-Bench与AVCap-Score，用于评估音视频字幕原子层面细节的专用基准与指标。我们的代码、模型和数据集可在该httpsURL获取。
英文摘要
Detailed audio-video joint captioning is essential for multimodal video understanding and generation. However, prior works are constrained by three main limitations: (1) the scarcity of high-quality public datasets with fine-grained audio-visual joint captions; (2) reinforcement-learning methods that rely on coarse reward signals; and (3) the lack of a benchmark and metric for evaluating detailed audiovisual captions at the atomic level. To address these challenges, we propose: (1) AVCap-100K, a high-quality dataset of 100K temporally aligned, detail-rich audio-video captions; (2) AVCap, a model optimized via Detail-Aware GRPO (Da-GRPO) that achieves state-of-the-art performance among open-source models and matches or surpasses proprietary models on several evaluations; and (3) AVCap-Bench and AVCap-Score, a specialized benchmark and metric for evaluating atomic-level details in audiovisual captions. Our code, models, and datasets are available at https://huggingface.co/collections/Apryle/avcap.
