DreamJourney：基于视频扩散模型的永续视角生成
DreamJourney: Perpetual View Generation with Video Diffusion Models
AI总结：
DreamJourney提出两阶段框架，利用视频扩散模型从单图生成含相机运动和物体动态的永续视角视频，通过3D点云提升、跨视角补全及多模态提示动画，显著优于现有方法。
摘要：
永续视角生成旨在仅从单张输入图像出发，合成与任意相机轨迹相对应的长时视频。现有方法通常利用预训练的文本到图像扩散模型，沿相机运动合成先前未见区域的新内容。然而，底层2D扩散模型缺乏3D感知，导致产生畸变伪影。此外，这些方法仅限于生成静态3D场景的视角，忽略了动态4D世界中物体运动的捕捉。为缓解这些问题，我们提出DreamJourney，一个两阶段框架，利用视频扩散模型的世界模拟能力，触发同时包含相机运动和物体动态的永续场景视角生成新任务。具体而言，在阶段I中，DreamJourney首先将输入图像提升为3D点云，并从特定相机轨迹渲染一系列部分图像。随后利用视频扩散模型作为生成先验，补全缺失区域并增强序列间的视觉连贯性，生成符合3D场景和相机轨迹的跨视角一致视频。同时，我们引入两种简单而有效的策略（早停和视角填充）以进一步稳定生成过程并提升视觉质量。接下来，在阶段II中，DreamJourney利用多模态大语言模型生成描述当前视角中物体运动的文本提示，并使用视频扩散模型对当前视角进行物体运动动画化。阶段I和阶段II循环重复，实现永续动态场景视角生成。大量实验在定量和定性上均证明了我们的DreamJourney相较于最先进方法的优越性。我们的项目页面：https://dream-journey.vercel.app。
英文摘要：
Perpetual view generation aims to synthesize a long-term video corresponding to an arbitrary camera trajectory solely from a single input image. Recent methods commonly utilize a pre-trained text-to-image diffusion model to synthesize new content of previously unseen regions along camera movement. However, the underlying 2D diffusion model lacks 3D awareness and results in distorted artifacts. Moreover, they are limited to generating views of static 3D scenes, neglecting to capture object movements within the dynamic 4D world. To alleviate these issues, we present DreamJourney, a two-stage framework that leverages the world simulation capacity of video diffusion models to trigger a new perpetual scene view generation task with both camera movements and object dynamics. Specifically, in stage I, DreamJourney first lifts the input image to 3D point cloud and renders a sequence of partial images from a specific camera trajectory. A video diffusion model is then utilized as generative prior to complete the missing regions and enhance visual coherence across the sequence, producing a cross-view consistent video adheres to the 3D scene and camera trajectory. Meanwhile, we introduce two simple yet effective strategies (early stopping and view padding) to further stabilize the generation process and improve visual quality. Next, in stage II, DreamJourney leverages a multimodal large language model to produce a text prompt describing object movements in current view, and uses video diffusion model to animate current view with object movements. Stage I and II are repeated recurrently, enabling perpetual dynamic scene view generation. Extensive experiments demonstrate the superiority of our DreamJourney over state-of-the-art methods both quantitatively and qualitatively. Our project page: https://dream-journey.vercel.app.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
