arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.30855cs.CV

Robust Dreamer: 用于动作控制AR视频生成的偏差感知潜在高斯记忆

Robust Dreamer: Deviation-Aware Latent Gaussian Memory for Action-Controlled AR Video Generation

School of Computing, National University of Singapore(新加坡国立大学计算机学院);Technische Universität München(慕尼黑技术大学);Vertex Lab(Vertex实验室);Australian National University(澳大利亚国立大学)

Hanlin Chen, Jiaxin Wei, Xibin Song, Yifu Wang, Steve Wang, Hongdong Li, Pan Ji, Gim Hee Lee

AI总结:

提出Robust Dreamer框架,通过潜在高斯记忆和动态偏差存档解决自回归视频生成中的漂移问题,实现长程3D一致性。

摘要:

逐帧动作控制的图像到视频生成是交互式世界模拟的一种有前景的范式,其中每个控制信号应引发即时的视觉响应。然而,在长自回归展开中保持视觉保真度和3D一致性仍然具有挑战性。现有的3D感知方法常常因两个障碍而遭受灾难性漂移:来自 extit{潜在--RGB循环}的信息丢失,其中生成的潜在被反复解码为RGB并重新编码用于未来条件;以及由 extit{无误差假设}引起的训练--推理差距,其中干净的训练记忆无法匹配预测损坏的推理记忆。为了解决这些挑战,我们提出了 extbf{Robust Dreamer},这是一个围绕如何设计3D记忆以及如何稳健使用它而构建的记忆增强框架。首先,我们引入了 extbf{潜在高斯记忆},它将从生成过程中继承的扩散潜在锚定到高斯基元,并通过潜在空间高斯泼溅召回它们。这提供了密集、几何感知、视图对齐的条件,同时避免了重复VAE转换导致的累积退化。其次,我们提出了 extbf{带有动态偏差存档的偏差学习},它通过一步近似合成展开引起的潜在偏差,按自回归阶段和去噪时间戳存储,并在训练期间将其注入历史记忆。这使得生成器暴露于现实的损坏记忆状态,并在推理前学习内部修正。在ScanNet、DL3DV和OmniWorldGame上的实验证明了最先进的长程性能。

英文摘要:

Frame-wise action-controlled image-to-video generation is a promising paradigm for interactive world simulation, where each control signal should elicit an immediate visual response. However, maintaining visual fidelity and 3D consistency over long autoregressive rollouts remains challenging. Existing 3D-aware methods often suffer from catastrophic drift due to two impediments: information loss from \textit{Latent--RGB Cycling}, where generated latents are repeatedly decoded to RGB and re-encoded for future conditioning, and the training--inference gap induced by the \textit{error-free hypothesis}, where clean training memory fails to match prediction-corrupted inference memory. To address these challenges, we present \textbf{Robust Dreamer}, a memory-augmented framework built around how to design 3D memory and how to use it robustly. First, we introduce \textbf{Latent Gaussian Memory}, which anchors diffusion latents inherited from the generation process to Gaussian primitives and recalls them via latent-space Gaussian splatting. This provides dense, geometry-aware, view-aligned conditioning while avoiding accumulated degradation from repeated VAE conversion. Second, we propose \textbf{Deviation Learning with Dynamic Deviation Archive}, which synthesizes rollout-induced latent deviations through a one-step approximation, stores them by autoregressive stage and denoising timestamp, and injects them into historical memory during training. This exposes the generator to realistic corrupted memory states and teaches internal correction before inference. Experiments on ScanNet, DL3DV, and OmniWorldGame demonstrate state-of-the-art long-horizon performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑