arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.11929cs.CVcs.GR
收藏

DiLightNet:基于扩散的图像生成中的细粒度光照控制

DiLightNet: Fine-grained Lighting Control for Diffusion-based Image Generation

State Key Lab of CAD and CG, Zhejiang University(浙江大学CAD与CG国家重点实验室);Microsoft Research Asia(微软亚洲研究院);College of William & Mary(威廉玛丽学院);Tsinghua University(清华大学)

Chong Zeng, Yue Dong, Pieter Peers, Youkang Kong, Hongzhi Wu, Xin Tong

AI总结:

本文提出DiLightNet,一种三阶段扩散模型光照控制方法,利用辐射提示引导生成过程,实现细粒度光照控制,无需精确几何,保留纹理细节并保持背景一致。

摘要:

本文提出了一种在文本驱动的扩散式图像生成过程中实现细粒度光照控制的新方法。尽管现有的扩散模型已经能够在任意光照条件下生成图像,但在没有额外引导的情况下,这些模型往往将图像内容与光照相关联。此外,文本提示缺乏描述详细光照设置所需的表达能力。为了在图像生成过程中为内容创作者提供对光照的细粒度控制,我们以辐射提示的形式为文本提示补充详细的光照信息,即目标光照下具有均匀规范材质的场景几何体的可视化。然而,生成辐射提示所需的场景几何体是未知的。我们的关键观察是,我们只需要引导扩散过程,因此精确的辐射提示并非必要;我们只需将扩散模型指向正确的方向。基于这一观察,我们引入了一种三阶段方法,用于在图像生成过程中控制光照。在第一阶段,我们利用标准的预训练扩散模型在不受控光照下生成一个临时图像。接下来,在第二阶段,我们通过将目标光照传递给一个名为DiLightNet的精细化扩散模型,使用根据临时图像推断出的前景物体粗略形状计算出的辐射提示,重新合成并细化生成图像中的前景物体。为了保留纹理细节,我们在将辐射提示传递给DiLightNet之前,将其与临时合成图像的神经编码相乘。最后,在第三阶段,我们重新合成背景,使其与前景物体上的光照保持一致。我们在多种文本提示和光照条件下演示并验证了我们的光照控制扩散模型。

英文摘要:

This paper presents a novel method for exerting fine-grained lighting control during text-driven diffusion-based image generation. While existing diffusion models already have the ability to generate images under any lighting condition, without additional guidance these models tend to correlate image content and lighting. Moreover, text prompts lack the necessary expressional power to describe detailed lighting setups. To provide the content creator with fine-grained control over the lighting during image generation, we augment the text-prompt with detailed lighting information in the form of radiance hints, i.e., visualizations of the scene geometry with a homogeneous canonical material under the target lighting. However, the scene geometry needed to produce the radiance hints is unknown. Our key observation is that we only need to guide the diffusion process, hence exact radiance hints are not necessary; we only need to point the diffusion model in the right direction. Based on this observation, we introduce a three stage method for controlling the lighting during image generation. In the first stage, we leverage a standard pretrained diffusion model to generate a provisional image under uncontrolled lighting. Next, in the second stage, we resynthesize and refine the foreground object in the generated image by passing the target lighting to a refined diffusion model, named DiLightNet, using radiance hints computed on a coarse shape of the foreground object inferred from the provisional image. To retain the texture details, we multiply the radiance hints with a neural encoding of the provisional synthesized image before passing it to DiLightNet. Finally, in the third stage, we resynthesize the background to be consistent with the lighting on the foreground object. We demonstrate and validate our lighting controlled diffusion model on a variety of text prompts and lighting conditions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑