arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2607.02798cs.CVcs.GR
收藏

追踪噪声,移动世界:用于可控视频生成的3D地面运动一致噪声

Track the Noise, Move the World:3D-Grounded Motion-Consistent Noise for Controllable Video Generation

Qualcomm AI Research(高通人工智能研究中心);Trinity College Dublin(都柏林三一学院)

Long Vu, Tan Ngo, Animesh Karnewar, Amir Habibian, Binh-Son Hua, Hung Bui, Minh Hoai Nguyen, Phong Nguyen-Ha

AI总结:

研究如何在视频生成中实现对物体和相机运动的精确统一控制。核心方法是构建统一框架UniCaMo,直接构造扩散模型输入噪声。主要贡献是在视频质量和运动可控性上取得先进成果,且无需对基础模型做大改动。

摘要:

现代图像与文本到视频的扩散模型可通过对初始高斯噪声张量去噪合成逼真视频,但现有方法在单代过程中对物体和相机运动缺乏精确统一控制。我们提出UniCaMo,一个通过直接构建扩散模型输入噪声来同时控制物体轨迹和相机视点的统一框架。具体而言,UniCaMo构建跨潜在视频帧的共享3D地面运动一致噪声空间。稀疏3D点轨迹用于沿期望物体轨迹扭曲参考帧的高斯噪声,而虚拟球面噪声表示为相机运动下新露出的场景区域提供全局一致的噪声值。通过将局部轨迹引导的噪声扭曲与基于全局球面的噪声采样相结合,UniCaMo在物体移动和视点变化下都保持几何和时间一致性。由于UniCaMo仅修改输入噪声,因此无需辅助适配器、控制分支或对基础视频扩散模型进行架构更改。通过在包括Wan 2.1(14B)在内的大型预训练视频扩散模型上进行轻量级LoRA微调,UniCaMo在标准可控视频生成基准测试中在视频质量和运动可控性方面均取得了领先成果。

英文摘要:

Modern image-and-text-to-video diffusion models can synthesize highly realistic videos by iteratively denoising an initial Gaussian noise tensor conditioned on reference image and text inputs. However, existing approaches still lack precise and unified controllability over both object motion and camera motion within a single generation process. We present UniCaMo, a unified framework that enables simultaneous control of object trajectories and camera viewpoints by directly constructing the input noise of the diffusion model. Specifically, UniCaMo builds a shared 3D-grounded motion-consistent noise space across latent video frames. Sparse 3D point tracks are used to warp the Gaussian noise of the reference frame along desired object trajectories, while a virtual spherical noise representation provides globally consistent noise values for newly revealed scene regions under camera motion. By combining local track-guided noise warping with global sphere-based noise sampling, UniCaMo maintains geometric and temporal consistency under both object movement and viewpoint changes. Because UniCaMo modifies only the input noise, it requires no auxiliary adapters, control branches, or architectural changes to the underlying video diffusion model. With lightweight LoRA fine-tuning on large pretrained video diffusion models, including Wan 2.1 (14B), UniCaMo achieves state-of-the-art results in both video quality and motion controllability on standard controllable video generation benchmarks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑