SkyReels-V3 技术报告
SkyReels-V3 Technique Report
AI总结：
SkyReels-V3 提出三种视频生成范式，通过多模态上下文学习框架实现高质量视频生成，涵盖图像到视频、视频扩展和音频引导生成，达到接近闭源系统的性能。
摘要：
视频生成是构建世界模型的基础，其中多模态上下文推理是能力的决定性测试。为此，我们提出了SkyReels-V3，一种基于统一多模态上下文学习框架的条件视频生成模型，结合扩散Transformer。SkyReels-V3模型在单一架构中支持三种核心生成范式：参考图像到视频合成、视频到视频扩展和音频引导的视频生成。（i）参考图像到视频模型旨在生成高保真视频，具有强主体身份保持、时间一致性和叙事一致性。为增强参考贴合度和构图稳定性，我们设计了一个全面的数据处理流程，利用跨帧配对、图像编辑和语义重写，有效减轻了复制粘贴伪影。在训练过程中，采用图像视频混合策略结合多分辨率联合优化，以提高在多样化场景中的泛化能力和鲁棒性。（ii）视频扩展模型集成了时空一致性建模与大规模视频理解，能够实现无缝单次延续和智能多次切换，具有专业电影化模式。 （iii）说话人avatar模型通过训练首帧和末帧插入模式并重建关键帧推断范式，支持分钟级音频条件视频生成。在保证视觉质量的基础上，音频与视频的同步性已得到优化。广泛的评估表明，SkyReels-V3在视觉质量、指令遵循和特定方面指标上达到了最先进的或接近最先进的性能，接近领先的闭源系统。Github: https://github.com/SkyworkAI/SkyReels-V3。
英文摘要：
Video generation serves as a cornerstone for building world models, where multimodal contextual inference stands as the defining test of capability. In this end, we present SkyReels-V3, a conditional video generation model, built upon a unified multimodal in-context learning framework with diffusion Transformers. SkyReels-V3 model supports three core generative paradigms within a single architecture: reference images-to-video synthesis, video-to-video extension and audio-guided video generation. (i) reference images-to-video model is designed to produce high-fidelity videos with strong subject identity preservation, temporal coherence, and narrative consistency. To enhance reference adherence and compositional stability, we design a comprehensive data processing pipeline that leverages cross frame pairing, image editing, and semantic rewriting, effectively mitigating copy paste artifacts. During training, an image video hybrid strategy combined with multi-resolution joint optimization is employed to improve generalization and robustness across diverse scenarios. (ii) video extension model integrates spatio-temporal consistency modeling with large-scale video understanding, enabling both seamless single-shot continuation and intelligent multi-shot switching with professional cinematographic patterns. (iii) Talking avatar model supports minute-level audio-conditioned video generation by training first-and-last frame insertion patterns and reconstructing key-frame inference paradigms. On the basis of ensuring visual quality, synchronization of audio and videos has been optimized. Extensive evaluations demonstrate that SkyReels-V3 achieves state-of-the-art or near state-of-the-art performance on key metrics including visual quality, instruction following, and specific aspect metrics, approaching leading closed-source systems. Github: https://github.com/SkyworkAI/SkyReels-V3.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
