arXiv 2501.038472025-01-09cs.CVcs.AIcs.GR

Diffusion as Shader：面向通用视频生成控制的3D感知视频扩散模型

Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Control

Hong Kong University of Science and Technology（香港科技大学）；Zhejiang University（浙江大学）；The University of Hong Kong（香港大学）；Nanyang Technological University（南洋理工大学）；Wuhan University（武汉大学）；Texas A&M University（德克萨斯农工大学）

Zekai Gu, Rui Yan, Jiahao Lu, Peng Li, Zhiyang Dou, Chenyang Si, Zhen Dong, Qifeng Liu, Cheng Lin, Ziwei Liu, Wenping Wang, Yuan Liu

备注：: Project page: https://igl-hkust.github.io/das/ Codes: https://github.com/IGL-HKUST/DiffusionAsShader

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出统一的3D感知视频扩散框架DaS，以3D跟踪视频作为控制输入，支持相机控制、运动迁移、物体操控和网格到视频生成等多种任务，并提升时间一致性。

摘要：

扩散模型在根据文本提示或图像生成高质量视频方面已展现出令人瞩目的性能。然而，对视频生成过程进行精确控制，例如相机操控或内容编辑，仍然是一项重大挑战。现有的受控视频生成方法通常局限于单一控制类型，缺乏处理多样化控制需求的灵活性。本文提出 Diffusion as Shader（DaS），这是一种在统一架构内支持多种视频控制任务的新方法。我们的关键洞察是，实现通用视频控制需要利用3D控制信号，因为视频本质上是动态3D内容的2D渲染。与局限于2D控制信号的以往方法不同，DaS利用3D跟踪视频作为控制输入，使视频扩散过程具备内在的3D感知能力。这一创新使DaS能够通过简单操控3D跟踪视频实现广泛的视频控制。使用3D跟踪视频的另一个优势是其能够有效关联各帧，显著增强生成视频的时间一致性。仅在8块H800 GPU上使用不到1万个视频微调3天，DaS就在网格到视频生成、相机控制、运动迁移和物体操控等多种任务中展现出强大的控制能力。

英文摘要：

Diffusion models have demonstrated impressive performance in generating high-quality videos from text prompts or images. However, precise control over the video generation process, such as camera manipulation or content editing, remains a significant challenge. Existing methods for controlled video generation are typically limited to a single control type, lacking the flexibility to handle diverse control demands. In this paper, we introduce Diffusion as Shader (DaS), a novel approach that supports multiple video control tasks within a unified architecture. Our key insight is that achieving versatile video control necessitates leveraging 3D control signals, as videos are fundamentally 2D renderings of dynamic 3D content. Unlike prior methods limited to 2D control signals, DaS leverages 3D tracking videos as control inputs, making the video diffusion process inherently 3D-aware. This innovation allows DaS to achieve a wide range of video controls by simply manipulating the 3D tracking videos. A further advantage of using 3D tracking videos is their ability to effectively link frames, significantly enhancing the temporal consistency of the generated videos. With just 3 days of fine-tuning on 8 H800 GPUs using less than 10k videos, DaS demonstrates strong control capabilities across diverse tasks, including mesh-to-video generation, camera control, motion transfer, and object manipulation.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。