SkyReels-A2:在视频扩散Transformer中组合任意内容
SkyReels-A2: Compose Anything in Video Diffusion Transformers
AI总结:
本文提出SkyReels-A2框架,通过数据管道构建和图像-文本联合嵌入模型解决元素到视频(E2V)生成中的保真度与连贯性问题,并引入A2 Bench基准,实现精确元素控制,成为首个开源商业级E2V模型。
摘要:
本文提出了SkyReels-A2,一个可控视频生成框架,能够基于文本提示将任意视觉元素(如角色、对象、背景)组装到合成视频中,同时保持每个元素与参考图像的严格一致性。我们将此任务称为元素到视频(E2V),其主要挑战在于保持每个参考元素的保真度、确保场景的连贯组合以及实现自然输出。为此,我们首先设计了一个全面的数据管道,为模型训练构建提示-参考-视频三元组。接着,我们提出了一种新颖的图像-文本联合嵌入模型,将多元素表示注入生成过程,平衡元素特定的一致性与全局连贯性及文本对齐。我们还优化了推理管道,以提升速度和输出稳定性。此外,我们引入了一个精心策划的基准用于系统评估,即A2 Bench。实验表明,我们的框架能够生成多样、高质量的视频,并实现精确的元素控制。SkyReels-A2是首个用于E2V生成的开源商业级模型,其表现与先进的闭源商业模型相当。我们期待SkyReels-A2将推动如戏剧和虚拟电商等创意应用,拓展可控视频生成的边界。
英文摘要:
This paper presents SkyReels-A2, a controllable video generation framework capable of assembling arbitrary visual elements (e.g., characters, objects, backgrounds) into synthesized videos based on textual prompts while maintaining strict consistency with reference images for each element. We term this task elements-to-video (E2V), whose primary challenges lie in preserving the fidelity of each reference element, ensuring coherent composition of the scene, and achieving natural outputs. To address these, we first design a comprehensive data pipeline to construct prompt-reference-video triplets for model training. Next, we propose a novel image-text joint embedding model to inject multi-element representations into the generative process, balancing element-specific consistency with global coherence and text alignment. We also optimize the inference pipeline for both speed and output stability. Moreover, we introduce a carefully curated benchmark for systematic evaluation, i.e, A2 Bench. Experiments demonstrate that our framework can generate diverse, high-quality videos with precise element control. SkyReels-A2 is the first open-source commercial grade model for the generation of E2V, performing favorably against advanced closed-source commercial models. We anticipate SkyReels-A2 will advance creative applications such as drama and virtual e-commerce, pushing the boundaries of controllable video generation.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。