D-ORCA：面向鲁棒音频视觉描述的对话中心优化
D-ORCA: Dialogue-Centric Optimization for Robust Audio-Visual Captioning
AI总结：
D-ORCA通过对话中心优化，提升音频视觉描述的鲁棒性和准确性，填补开源生态关键空白。
摘要：
语音对话是视频中的主要信息来源；因此，准确识别谁说了什么以及何时说的至关重要，对于深度视频理解而言。我们介绍了D-ORCA，一种以对话为中心的多模态大语言模型，专门优化于鲁棒的音频视觉描述。我们进一步精心编纂了DVD，一个大规模、高质量的双语数据集，包含近40,000个多方对话视频用于训练，以及2000个视频用于英文和中文的评估，填补了开源生态中的关键空白。为了确保细粒度描述的准确性，我们采用群体相对策略优化，采用三种新的奖励函数，评估说话者归属准确性、全局语音内容准确性和句子层面的时间边界对齐。这些奖励来源于广泛用于语音处理的评估指标，并且据我们所知，首次应用于音频视觉描述的强化学习目标。大量实验表明，D-ORCA在说话人识别、语音识别和时间定位方面显著优于现有的开源模型。值得注意的是，尽管只有80亿参数，D-ORCA在几个通用的音频视觉理解基准上实现了与Qwen3-Omni相媲美的性能。演示可在https://d-orca-llm.github.io/上找到。我们的代码、数据和检查点将在https://github.com/WeChatCV/D-ORCA/上提供。
英文摘要：
Spoken dialogue is a primary source of information in videos; therefore, accurately identifying who spoke what and when is essential for deep video understanding. We introduce D-ORCA, a \textbf{d}ialogue-centric \textbf{o}mni-modal large language model optimized for \textbf{r}obust audio-visual \textbf{ca}ptioning. We further curate DVD, a large-scale, high-quality bilingual dataset comprising nearly 40,000 multi-party dialogue videos for training and 2000 videos for evaluation in English and Mandarin, addressing a critical gap in the open-source ecosystem. To ensure fine-grained captioning accuracy, we adopt group relative policy optimization with three novel reward functions that assess speaker attribution accuracy, global speech content accuracy, and sentence-level temporal boundary alignment. These rewards are derived from evaluation metrics widely used in speech processing and, to our knowledge, are applied for the first time as reinforcement learning objectives for audio-visual captioning. Extensive experiments demonstrate that D-ORCA substantially outperforms existing open-source models in speaker identification, speech recognition, and temporal grounding. Notably, despite having only 8 billion parameters, D-ORCA achieves performance competitive with Qwen3-Omni across several general-purpose audio-visual understanding benchmarks. Demos are available at \href{https://d-orca-llm.github.io/}{https://d-orca-llm.github.io/}. Our code, data, and checkpoints will be available at \href{https://github.com/WeChatCV/D-ORCA/}{https://github.com/WeChatCV/D-ORCA/}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
