TimeChat-Captioner: 用时间感知和结构化的音视频字幕脚本化多场景视频
TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captions
AI总结：
提出Omni密集描述任务，通过六维结构模式生成带时间戳的类脚本描述，构建OmniDCBench基准和SodaM评估指标，训练TimeChat-Captioner-7B模型，在音视频推理和时间定位任务上超越Gemini-2.5-Pro。
摘要：
本文提出Omni密集描述（Omni Dense Captioning）这一新任务，旨在生成连续、细粒度且结构化的音视频叙事，并带有明确的时间戳。为确保密集语义覆盖，我们引入一个六维结构模式来创建“类脚本”描述，使读者能够像电影剧本一样逐场景生动想象视频内容。为促进研究，我们构建了OmniDCBench——一个高质量、人工标注的基准，并提出了SodaM——一个统一的评估指标，用于评估时间感知的详细描述，同时减轻场景边界模糊性。此外，我们构建了训练数据集TimeChatCap-42K，并提出了TimeChat-Captioner-7B——一个通过SFT和GRPO结合任务特定奖励训练的强基线模型。大量实验表明，TimeChat-Captioner-7B达到了最先进的性能，超越了Gemini-2.5-Pro，其生成的密集描述显著提升了下游能力，包括音视频推理（DailyOmni和WorldSense）和时间定位（Charades-STA）。所有数据集、模型和代码均可在该https URL获取。
英文摘要：
This paper proposes Omni Dense Captioning, a novel task designed to generate continuous, fine-grained, and structured audio-visual narratives with explicit timestamps. To ensure dense semantic coverage, we introduce a six-dimensional structural schema to create "script-like" captions, enabling readers to vividly imagine the video content scene by scene, akin to a cinematographic screenplay. To facilitate research, we construct OmniDCBench, a high-quality, human-annotated benchmark, and propose SodaM, a unified metric that evaluates time-aware detailed descriptions while mitigating scene boundary ambiguity. Furthermore, we construct a training dataset, TimeChatCap-42K, and present TimeChat-Captioner-7B, a strong baseline trained via SFT and GRPO with task-specific rewards. Extensive experiments demonstrate that TimeChat-Captioner-7B achieves state-of-the-art performance, surpassing Gemini-2.5-Pro, while its generated dense descriptions significantly boost downstream capabilities in audio-visual reasoning (DailyOmni and WorldSense) and temporal grounding (Charades-STA). All datasets, models, and code are available at https://github.com/yaolinli/TimeChat-Captioner.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
