OmniScript: 向长篇影视视频的音频-视觉脚本生成迈进
OmniScript: Towards Audio-Visual Script Generation for Long-Form Cinematic Video
AI总结：
本文提出视频到脚本任务，介绍OmniScript模型，通过渐进式训练在长篇叙事理解中实现高效脚本生成，优于开源模型并接近专有模型。
摘要：
当前多模态大语言模型在短视频理解方面表现出色，但将长篇影视视频转化为详细、时间相关的脚本仍具挑战性。本文引入视频到脚本（V2S）任务，旨在生成分场景的分层脚本，涵盖角色动作、对话、表情和音频提示。为此，我们构建了首个人工标注的基准，并提出一种时间感知的分层评估框架。此外，我们提出了OmniScript，一个80亿参数的多模态（音频-视觉）语言模型，专门用于长篇叙述理解。OmniScript通过渐进式管道训练，利用链式思维监督微调进行情节和角色推理，随后通过时间分段奖励进行强化学习。大量实验表明，尽管参数高效，OmniScript在时间定位和多字段语义准确性方面显著优于更大的开源模型，并在Gemini 3-Pro等顶级专有模型中表现相当。
英文摘要：
Current multimodal large language models (MLLMs) have demonstrated remarkable capabilities in short-form video understanding, yet translating long-form cinematic videos into detailed, temporally grounded scripts remains a significant challenge. This paper introduces the novel video-to-script (V2S) task, aiming to generate hierarchical, scene-by-scene scripts encompassing character actions, dialogues, expressions, and audio cues. To facilitate this, we construct a first-of-its-kind human-annotated benchmark and propose a temporally-aware hierarchical evaluation framework. Furthermore, we present OmniScript, an 8B-parameter omni-modal (audio-visual) language model tailored for long-form narrative comprehension. OmniScript is trained via a progressive pipeline that leverages chain-of-thought supervised fine-tuning for plot and character reasoning, followed by reinforcement learning using temporally segmented rewards. Extensive experiments demonstrate that despite its parameter efficiency, OmniScript significantly outperforms larger open-source models and achieves performance comparable to state-of-the-art proprietary models, including Gemini 3-Pro, in both temporal localization and multi-field semantic accuracy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
