RGBX-Next：基于G-buffers的真实感生成式渲染
RGBX-Next: Towards Realistic Generative Rendering from G-Buffers
- University of California, Santa Barbara（加利福尼亚大学圣巴巴拉分校）
- NVIDIA（英伟达公司）
- University of California, San Diego（加利福尼亚大学圣迭戈分校）
- Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
本文提出RGBX-Next框架，将扩散Transformer（DiT）微调为正向与反向渲染器，可从图像等估算G-buffers或从G-buffers渲染真实感内容，在相关任务中表现优异，将公开模型并推动领域研究。
AI中文摘要：
扩散模型在图像、视频及流生成领域已取得令人瞩目的成果，但与传统3D渲染相比，其对生成输出的精确控制仍有所欠缺。我们认为可行的发展路径是将生成模型作为基于传统渲染得到的G-buffers的学习型渲染器。本文提出RGBX-Next，这是一个适用于正向与反向渲染的统一生成框架，支持从图像、视频及流中估算G-buffers，也可从G-buffers渲染出真实感图像、视频及流。我们的核心贡献是提出了将扩散Transformer（DiT）模型微调为生成式正向与反向渲染器的通用方案。实验表明，所得模型在真实感生成式渲染与内在分解任务中均达到了高质量表现。我们将公开所有模型，相信本文提出的设计原则将有益于未来可控生成式正向与反向渲染的研究。
英文摘要：
Diffusion models have achieved impressive results in image, video, and streaming generation. However, compared to traditional 3D rendering, they still lack precise control over the generated output. We believe a viable path forward is to use generative models as learned renderers conditioned on traditionally rendered G-buffers. We introduce RGBX-Next, a unified generative framework for forward and inverse rendering, which allows estimating G-buffers from images, videos, and streams, and rendering realistic images, videos, and streams from G-buffers. Our key contribution is a general recipe for finetuning diffusion transformer (DiT) models into generative forward and inverse renderers. We show that the resulting models achieve high quality in both realistic generative rendering and intrinsic decomposition. We will make all our models publicly available. We believe that the design principles presented in this paper will benefit future research on controllable generative forward and inverse rendering.
