Tele-Omni: 一种用于视频生成与编辑的统一多模态框架
Tele-Omni: a Unified Multimodal Framework for Video Generation and Editing
AI总结：
Tele-Omni是一种统一多模态框架，通过解析文本、图像和参考视频指令，实现视频生成与编辑的灵活控制，提升时间一致性和视觉一致性。
摘要：
最近基于扩散的视频生成技术显著提高了视觉保真度和时间一致性。然而，大多数现有方法仍然是任务特定的，并主要依赖于文本指令，限制了它们在统一框架中处理多模态输入、上下文参考和多样化的视频生成和编辑场景的能力。此外，许多视频编辑方法依赖于精心设计的管道，专门针对个别操作，这阻碍了可扩展性和可组合性。在本文中，我们提出了Tele-Omni，一种用于视频生成和编辑的统一多模态框架，该框架遵循多模态指令，包括文本、图像和参考视频，在单个模型中。Tele-Omni利用预训练的多模态大语言模型来解析异构指令并推断结构化的生成或编辑意图，同时基于扩散的生成器根据这些结构化信号进行高质量视频合成。为了在异构视频任务之间实现联合训练，我们引入了一个任务感知的数据处理流程，将多模态输入统一为结构化指令格式，同时保持任务特定的约束。Tele-Omni支持一系列视频中心的任务，包括文本到视频生成、图像到视频生成、首尾帧视频生成、上下文视频生成和上下文视频编辑。通过将指令解析与视频合成解耦，并结合任务感知的数据设计，Tele-Omni实现了灵活的多模态控制，同时保持强大的时间一致性和视觉一致性。实验结果表明，Tele-Omni在多个任务上实现了竞争性的性能。
英文摘要：
Recent advances in diffusion-based video generation have substantially improved visual fidelity and temporal coherence. However, most existing approaches remain task-specific and rely primarily on textual instructions, limiting their ability to handle multimodal inputs, contextual references, and diverse video generation and editing scenarios within a unified framework. Moreover, many video editing methods depend on carefully engineered pipelines tailored to individual operations, which hinders scalability and composability. In this paper, we propose Tele-Omni, a unified multimodal framework for video generation and editing that follows multimodal instructions, including text, images, and reference videos, within a single model. Tele-Omni leverages pretrained multimodal large language models to parse heterogeneous instructions and infer structured generation or editing intents, while diffusion-based generators perform high-quality video synthesis conditioned on these structured signals. To enable joint training across heterogeneous video tasks, we introduce a task-aware data processing pipeline that unifies multimodal inputs into a structured instruction format while preserving task-specific constraints. Tele-Omni supports a wide range of video-centric tasks, including text-to-video generation, image-to-video generation, first-last-frame video generation, in-context video generation, and in-context video editing. By decoupling instruction parsing from video synthesis and combining it with task-aware data design, Tele-Omni achieves flexible multimodal control while maintaining strong temporal coherence and visual consistency. Experimental results demonstrate that Tele-Omni achieves competitive performance across multiple tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
