arXiv 2602.218182026-03-18cs.CV

SkyReels-V4：多模态视频-音频生成、修复和编辑模型

SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model

Skywork AI

Guibin Chen, Dixuan Lin, Jiangping Yang, Youqiang Zhang, Zhengcong Fei, Debang Li, Sheng Chen, Chaofeng Ao, Nuo Pang, Yiming Wang, Yikun Dou, Zheng Chen, Mingyu… 展开作者

Guibin Chen, Dixuan Lin, Jiangping Yang, Youqiang Zhang, Zhengcong Fei, Debang Li, Sheng Chen, Chaofeng Ao, Nuo Pang, Yiming Wang, Yikun Dou, Zheng Chen, Mingyuan Fan, Tuanhui Li, Mingshan Chang, Hao Zhang, Xiaopeng Sun, Jingtao Xu, Yuqiang Xie, Jiahua Wang, Zhiheng Xu, Weiming Xiong, Yuzhe Jin, Baoxuan Gu, Binjie Mao, Yunjie Yu, Jujie He, Yuhao Feng, Shiwen Tu, Chaojie Wang, Rui Yan, Wei Shen, Jingchen Wu, Peng Zhao, Xuanyue Zhong, Zhuangzhuang Liu, Kaifei Wang, Fuxiang Zhang, Weikai Xu, Wenyan Liu, Binglu Zhang, Yu Shen, Tianhui Xiong, Bin Peng, Liang Zeng, Xuchen Song, Haoxiang Guo, Peiyu Wang, Max W. Y. Lam, Chien-Hung Liu, Yahui Zhou

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

SkyReels-V4是一款多模态视频基础模型，支持视频音频生成、修复和编辑，采用双流多模态扩散变换器架构，结合文本编码器实现多模态指令处理，支持高分辨率、长时长视频生成。

摘要：

SkyReels V4 是一个统一的多模态视频基础模型，用于联合视频音频生成、修复和编辑。该模型采用双流多模态扩散变换器（MMDiT）架构，其中一路合成视频，另一路生成时间对齐的音频，同时共享基于多模态大语言模型（MLLM）的强大文本编码器。SkyReels V4 接受丰富的多模态指令，包括文本、图像、视频片段、掩码和音频参考。通过结合 MLLM 的多模态指令跟随能力与视频分支 MMDiT 的上下文学习，模型能够在复杂条件下注入精细的视觉指导，同时音频分支 MMDiT 利用音频参考指导声音生成。在视频侧，我们采用通道拼接公式，统一了多种修复风格任务，如图像到视频、视频扩展和视频编辑，通过多模态提示自然扩展到基于视觉的修复和编辑。SkyReels V4 支持高达 1080p 分辨率、32 FPS 和 15 秒时长，实现高保真、多镜头、电影级视频生成。为使高分辨率、长时长生成计算可行，我们引入了效率策略：联合生成低分辨率完整序列和高分辨率关键帧，随后使用专用超分辨率和帧插值模型。据我们所知，SkyReels V4 是首个同时支持多模态输入、联合视频音频生成以及生成、修复和编辑统一处理的视频基础模型，同时在电影级分辨率和时长下保持强大的效率和质量。

英文摘要：

SkyReels V4 is a unified multi modal video foundation model for joint video audio generation, inpainting, and editing. The model adopts a dual stream Multimodal Diffusion Transformer (MMDiT) architecture, where one branch synthesizes video and the other generates temporally aligned audio, while sharing a powerful text encoder based on the Multimodal Large Language Models (MLLM). SkyReels V4 accepts rich multi modal instructions, including text, images, video clips, masks, and audio references. By combining the MLLMs multi modal instruction following capability with in context learning in the video branch MMDiT, the model can inject fine grained visual guidance under complex conditioning, while the audio branch MMDiT simultaneously leverages audio references to guide sound generation. On the video side, we adopt a channel concatenation formulation that unifies a wide range of inpainting style tasks, such as image to video, video extension, and video editing under a single interface, and naturally extends to vision referenced inpainting and editing via multi modal prompts. SkyReels V4 supports up to 1080p resolution, 32 FPS, and 15 second duration, enabling high fidelity, multi shot, cinema level video generation with synchronized audio. To make such high resolution, long-duration generation computationally feasible, we introduce an efficiency strategy: Joint generation of low resolution full sequences and high-resolution keyframes, followed by dedicated super-resolution and frame interpolation models. To our knowledge, SkyReels V4 is the first video foundation model that simultaneously supports multi-modal input, joint video audio generation, and a unified treatment of generation, inpainting, and editing, while maintaining strong efficiency and quality at cinematic resolutions and durations.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。