Script-a-Video: 通过因子化流和关系 grounding 实现深度结构化音频视觉描述
Script-a-Video: Deep Structured Audio-visual Captions via Factorized Streams and Relational Grounding
AI总结：
本文提出MTSS框架，通过因子化流和关系 grounding 解决视频描述中信息耦合问题，提升视频理解与生成性能，实验显示在多个基准测试中均取得显著改进。
摘要：
多模态大语言模型（MLLMs）的进步正将视频描述从描述性终点转变为视频理解和生成的语义接口。然而，主流范式仍将视频视为包含视觉、听觉和身份信息的单一叙事段落。这种紧密耦合不仅损害了表示保真度，还限制了可扩展性，因为即使局部编辑也可能触发全局重写。为解决这一结构瓶颈，我们提出了多流场景脚本（MTSS），一种新的范式，用因子化和显式接地的场景描述替代单一文本。MTSS基于两个核心原则：流因子化，将视频分解为互补的流（参考、镜头、事件和全局），以及关系接地，通过显式身份和时间链接重新连接这些孤立流，以保持整体视频一致性。广泛实验表明，MTSS在各种模型上均提升了视频理解能力，实现了在Video-SALMONN-2上的总错误率平均减少25%，在Daily-Omni推理基准上的平均性能提升67%。它还缩小了小型和大型MLLMs之间的性能差距，表明了一个显著更可学习的描述接口。最后，即使不进行架构调整，将单一提示替换为MTSS在多镜头视频生成中也产生了显著的人类评分改进：跨镜头身份一致性提高了45%，视听对齐提高了56%，时间可控性提高了71%。
英文摘要：
Advances in Multimodal Large Language Models (MLLMs) are transforming video captioning from a descriptive endpoint into a semantic interface for both video understanding and generation. However, the dominant paradigm still casts videos as monolithic narrative paragraphs that entangle visual, auditory, and identity information. This dense coupling not only compromises representational fidelity but also limits scalability, since even local edits can trigger global rewrites. To address this structural bottleneck, we propose Multi-Stream Scene Script (MTSS), a novel paradigm that replaces monolithic text with factorized and explicitly grounded scene descriptions. MTSS is built on two core principles: Stream Factorization, which decouples a video into complementary streams (Reference, Shot, Event, and Global), and Relational Grounding, which reconnects these isolated streams through explicit identity and temporal links to maintain holistic video consistency. Extensive experiments demonstrate that MTSS consistently enhances video understanding across various models, achieving an average reduction of 25% in the total error rate on Video-SALMONN-2 and an average performance gain of 67% on the Daily-Omni reasoning benchmark. It also narrows the performance gap between smaller and larger MLLMs, indicating a substantially more learnable caption interface. Finally, even without architectural adaptation, replacing monolithic prompts with MTSS in multi-shot video generation yields substantial human-rated improvements: a 45% boost in cross-shot identity consistency, a 56% boost in audio-visual alignment, and a 71% boost in temporal controllability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
