arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2309.02119cs.CV
收藏

用于视频外扩的分层掩码3D扩散模型

Hierarchical Masked 3D Diffusion Model for Video Outpainting

Alibaba Group(阿里巴巴集团);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学)

Fanda Fan, Chaoxu Guo, Litong Gong, Biao Wang, Tiezheng Ge, Yuning Jiang, Chunjie Luo, Jianfeng Zhan

AI总结:

本文提出一种掩码3D扩散模型用于视频外扩,通过掩码建模、全局帧提示和混合由粗到细推理流程,提升时间一致性并缓解伪影累积,取得最先进结果。

摘要:

视频外扩旨在充分补全视频帧边缘的缺失区域。与图像外扩相比,它提出了额外的挑战,因为模型需要保持填充区域的时间一致性。本文提出了一种用于视频外扩的掩码3D扩散模型。我们利用掩码建模技术训练3D扩散模型,从而能够使用多个引导帧连接多个视频片段推理的结果,确保时间一致性并减少相邻帧之间的抖动。同时,我们提取视频的全局帧作为提示,并通过交叉注意力引导模型获取当前视频片段之外的信息。我们还引入了一种混合的由粗到细的推理流程,以缓解伪影累积问题。现有的由粗到细流程仅使用填充策略,由于稀疏帧的时间间隔过大而导致性能退化。我们的流程受益于掩码建模的双向学习,因此在生成稀疏帧时可以采用填充与插值相结合的混合策略。实验表明,我们的方法在视频外扩任务中取得了最先进的结果。更多结果和代码见我们的 https://fanfanda.github.io/M3DDM/。

英文摘要:

Video outpainting aims to adequately complete missing areas at the edges of video frames. Compared to image outpainting, it presents an additional challenge as the model should maintain the temporal consistency of the filled area. In this paper, we introduce a masked 3D diffusion model for video outpainting. We use the technique of mask modeling to train the 3D diffusion model. This allows us to use multiple guide frames to connect the results of multiple video clip inferences, thus ensuring temporal consistency and reducing jitter between adjacent frames. Meanwhile, we extract the global frames of the video as prompts and guide the model to obtain information other than the current video clip using cross-attention. We also introduce a hybrid coarse-to-fine inference pipeline to alleviate the artifact accumulation problem. The existing coarse-to-fine pipeline only uses the infilling strategy, which brings degradation because the time interval of the sparse frames is too large. Our pipeline benefits from bidirectional learning of the mask modeling and thus can employ a hybrid strategy of infilling and interpolation when generating sparse frames. Experiments show that our method achieves state-of-the-art results in video outpainting tasks. More results and codes are provided at our https://fanfanda.github.io/M3DDM/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑