arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.11697cs.CV
收藏

MVTokenFlow:使用多视角 Token 流进行高质量 4D 内容生成

MVTokenFlow: High-quality 4D Content Generation using Multiview Token Flow

ShanghaiTech University(上海科技大学);The Hong Kong University of Science and Technology(香港科技大学);The University of Hong Kong(香港大学)

Hanzhuo Huang, Yuan Liu, Ge Zheng, Jiepeng Wang, Zhiyang Dou, Sibei Yang

AI总结:

提出 MVTokenFlow,通过多视角扩散构建粗略 4D 场,并以渲染 2D 流引导跨时间步复用 token 重生成一致图像,显著提升单目视频 4D 内容质量。

摘要:

本文提出 MVTokenFlow,用于从单目视频创建高质量 4D 内容。视频扩散模型和多视角扩散模型等生成模型的最新进展,使我们能够创建视频或 3D 模型。然而,将这些生成模型扩展到动态 4D 内容创建仍是一项具有挑战性的任务,因为所生成内容需要在空间和时间上保持一致。为应对这一挑战,MVTokenFlow 利用多视角扩散模型在不同时间步生成多视角图像,从而获得跨不同视角的空间一致性,并使我们能够重建合理的粗略 4D 场。随后,MVTokenFlow 进一步以渲染得到的 2D 流为引导,重新生成所有多视角图像。这些 2D 流能够有效关联来自不同时间步的像素,并通过在重新生成过程中复用 token 来提升时间一致性。最终,重新生成的图像具备时空一致性,并被用于细化粗略 4D 场,以获得高质量 4D 场。实验证明了我们设计的有效性,并显示其质量相较基线方法有显著提升。

英文摘要:

In this paper, we present MVTokenFlow for high-quality 4D content creation from monocular videos. Recent advancements in generative models such as video diffusion models and multiview diffusion models enable us to create videos or 3D models. However, extending these generative models for dynamic 4D content creation is still a challenging task that requires the generated content to be consistent spatially and temporally. To address this challenge, MVTokenFlow utilizes the multiview diffusion model to generate multiview images on different timesteps, which attains spatial consistency across different viewpoints and allows us to reconstruct a reasonable coarse 4D field. Then, MVTokenFlow further regenerates all the multiview images using the rendered 2D flows as guidance. The 2D flows effectively associate pixels from different timesteps and improve the temporal consistency by reusing tokens in the regeneration process. Finally, the regenerated images are spatiotemporally consistent and utilized to refine the coarse 4D field to get a high-quality 4D field. Experiments demonstrate the effectiveness of our design and show significantly improved quality than baseline methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑