CodecCap: 高保真度编解码器启发的残差建模用于密集视频字幕生成
CodecCap: High-Fidelity Codec-Inspired Residual Modeling for Dense Video Captioning
AI总结：
提出CodecCap框架，通过关键帧和残差字幕模拟视频编解码器，在保持细粒度视觉证据的同时减少冗余，并引入VidCapQA基准验证其高保真度。
摘要：
现有的视频字幕方法难以平衡视觉保真度和冗余：整体字幕紧凑但丢失细粒度证据，而分段字幕改善覆盖但引入大量冗余。我们提出CodecCap，一种受编解码器启发的高保真度密集视频字幕框架。类似于视频编解码器，CodecCap使用关键帧和残差字幕表示视频。关键帧字幕详尽编码稳定的视觉上下文，而残差字幕仅捕获时间上局部的动作、运动和变化。这有效保留了细粒度视觉证据，同时减少冗余描述。为了量化字幕的保真度，我们引入VidCapQA，一个包含14个能力维度1000个问题的字幕-问答基准。VidCapQA上的结果表明，强VLM直接生成的字幕仍然遗漏许多视觉细节，突显字幕表示是关键瓶颈。实验表明，CodecCap显著超越使用相同底层VLM的直接字幕生成，表明关键帧-残差字幕是一种高保真度视频-语言监督的方式。我们进一步使用CodecCap构建CodecVDC-100K，一个包含锚点、残差、场景级和视频级监督的大规模密集字幕数据集。
英文摘要：
Existing video captioning methods struggle to balance visual fidelity and redundancy: holistic captions are compact but lose fine-grained evidence, whereas segment-wise captions improve coverage but introduce heavy redundancy. We propose CodecCap, a codec-inspired framework for high-fidelity dense video captioning. Analogous to video codecs, CodecCap represents videos using keyframe and residual captions. Keyframe captions exhaustively encode stable visual context, while residual captions capture temporally only localized actions, motions and changes. This effectively preserves fine-grained visual evidence while reducing redundant descriptions. To quantify the fidelity of captions, we introduce VidCapQA, a caption-then-QA benchmark with 1,000 questions across 14 capability dimensions. Results on VidCapQA show that captions directly generated by strong VLMs still miss many visual details, highlighting caption representation as a critical bottleneck. Experiments show that CodecCap significantly surpasses direct captioning with the same underlying VLMs, suggesting keyframe-residual captioning a way for high-fidelity video-language supervision. We further use CodecCap to construct CodecVDC-100K, a large-scale dense captioning dataset with anchor, residual, scene-level, and video-level supervision.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
