GenLit：将单图像重照明重构为视频生成
GenLit: Reformulating Single-Image Relighting as Video Generation
AI总结：
GenLit将单图像重照明重构为视频生成，利用视频扩散模型隐式理解3D场景，蒸馏图形引擎能力，实现无需显式重建或光线追踪的逼真重照明。
摘要：
在单张图像内操控3D场景的照明是计算机视觉和图形学中的一个基本挑战。传统上，这一问题通过逆渲染技术解决，涉及显式的3D资产重建和昂贵的光线追踪模拟。与此同时，视觉基础模型的最新进展表明，一种新范式可能即将成为可能——即用在大规模图像和视频数据上训练的网络取代显式物理模型。在本文中，我们利用视频扩散模型（特别是Stable Video Diffusion）的隐式场景理解来对单张图像进行重照明。我们提出了GenLit，一个将图形引擎的光照操作能力蒸馏到视频生成模型中的框架，使用户能够直接在给定图像的3D世界中插入和操控点光源，并直接生成视频序列作为结果。我们发现，仅在小规模合成数据集上微调的模型能够泛化到真实世界场景，实现具有合理且令人信服的阴影和相互反射的单图像重照明。我们的结果凸显了视频基础模型捕捉丰富光照、材质和形状信息的能力，并表明此类模型经过最少训练即可用于重照明，无需显式资产重建或光线追踪。项目页面：https://genlit.is.tue.mpg.de/。
英文摘要：
Manipulating the illumination of a 3D scene within a single image represents a fundamental challenge in computer vision and graphics. This problem has traditionally been addressed using inverse rendering techniques, which involve explicit 3D asset reconstruction and costly ray-tracing simulations. Meanwhile, recent advancements in visual foundation models suggest that a new paradigm could soon be possible -- one that replaces explicit physical models with networks that are trained on large amounts of image and video data. In this paper, we exploit the implicit scene understanding of a video diffusion model, particularly Stable Video Diffusion, to relight a single image. We introduce GenLit, a framework that distills the ability of a graphics engine to perform light manipulation into a video-generation model, enabling users to directly insert and manipulate a point light in the 3D world within a given image and generate results directly as a video sequence. We find that a model fine-tuned on only a small synthetic dataset generalizes to real-world scenes, enabling single-image relighting with plausible and convincing shadows and inter-reflections. Our results highlight the ability of video foundation models to capture rich information about lighting, material, and shape, and our findings indicate that such models, with minimal training, can be used to perform relighting without explicit asset reconstruction or ray-tracing. . Project page: https://genlit.is.tue.mpg.de/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
