LoomVideo: 统一多模态输入到视频生成与编辑
LoomVideo: Unifying Multimodal Inputs into Video Generation and Editing
AI总结：
提出LoomVideo，一种5B参数的高效统一架构，通过多模态大语言模型和零开销Scale-and-Add条件机制，实现视频生成与编辑，显著降低计算复杂度并加速推理。
摘要：
开发能够解释交错多模态输入的统一视频生成和编辑模型是一个有前景但充满挑战的前沿领域。现有的统一框架主要依赖大规模模型（通常为13B参数或更多），并通过拼接序列令牌来引入源视频条件进行编辑。这种拼接不可避免地使序列长度加倍，使自注意力机制的计算复杂度翻两番，带来难以承受的开销。为解决这些瓶颈，我们提出了LoomVideo，一种高效5B参数的统一架构，用于视频生成和编辑。LoomVideo用多模态大语言模型（MLLM）替换标准文本编码器，并采用Deepstack注入机制将多层MLLM特征与扩散变换器（DiT）对齐。关键地，我们引入了一种零开销的Scale-and-Add条件方法用于视频编辑。通过缩放并直接将干净源视频潜变量加到带噪目标潜变量上，这种优雅的设计消除了令牌拼接的需要，大幅降低计算成本，同时保持对复杂非刚性编辑的强大能力。此外，无缝集成了负时间RoPE策略以处理多个参考图像。大量实验表明，我们紧凑的5B模型在全面基准测试中达到了最先进或极具竞争力的性能，在电商和时尚生成场景中展现出卓越优势。得益于零开销条件机制，LoomVideo在推理速度上比类似能力的模型至少快5.41倍，为高度实用和高效的视频基础模型铺平了道路。
英文摘要：
Developing unified video generation and editing models capable of interpreting interleaved multimodal inputs is a promising yet challenging frontier field. Existing unified frameworks predominantly rely on massive models (typically 13B parameters or more) and incorporate source video conditions for editing by concatenating sequence tokens. This concatenation inevitably doubles the sequence length, quadrupling the computational complexity of the self-attention mechanism and introducing prohibitive overhead. To address these bottlenecks, we present LoomVideo, a highly efficient 5B-parameter unified architecture for both video generation and editing. LoomVideo replaces the standard text encoder with a Multimodal Large Language Model (MLLM) and employs Deepstack injection mechanism to align multi-layer MLLM features with the Diffusion Transformer (DiT). Crucially, we introduce a zero-overhead Scale-and-Add conditioning approach for video editing. By scaling and directly adding the clean source video latent to the noised target latent, this elegant design eliminates the need for token concatenation, drastically reducing computational cost while maintaining robust capabilities for complex, non-rigid edits. Furthermore, a Negative Temporal RoPE strategy is seamlessly integrated to handle multiple reference images. Extensive experiments demonstrate that our compact 5B model achieves state-of-the-art or highly competitive performance across comprehensive benchmarks, exhibiting exceptional superiority in e-commerce and fashion generation scenarios. Benefiting from the zero-overhead conditioning mechanism, LoomVideo achieves at least a 5.41x acceleration in inference speed compared to models of similar capabilities, paving the way for highly practical and efficient video foundation models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
