DiLightNet: Fine-grained Lighting Control for Diffusion-based Image Generation
State Key Lab of CAD and CG, Zhejiang University(浙江大学CAD与CG国家重点实验室);Microsoft Research Asia(微软亚洲研究院);College of William & Mary(威廉玛丽学院);Tsinghua University(清华大学)
This paper presents a novel method for exerting fine-grained lighting control during text-driven diffusion-based image generation. While existing diffusion models already have the ability to generate images under any lighting condition, without additional guidance these models tend to correlate image content and lighting. Moreover, text prompts lack the necessary expressional power to describe detailed lighting setups. To provide the content creator with fine-grained control over the lighting during image generation, we augment the text-prompt with detailed lighting information in the form of radiance hints, i.e., visualizations of the scene geometry with a homogeneous canonical material under the target lighting. However, the scene geometry needed to produce the radiance hints is unknown. Our key observation is that we only need to guide the diffusion process, hence exact radiance hints are not necessary; we only need to point the diffusion model in the right direction. Based on this observation, we introduce a three stage method for controlling the lighting during image generation. In the first stage, we leverage a standard pretrained diffusion model to generate a provisional image under uncontrolled lighting. Next, in the second stage, we resynthesize and refine the foreground object in the generated image by passing the target lighting to a refined diffusion model, named DiLightNet, using radiance hints computed on a coarse shape of the foreground object inferred from the provisional image. To retain the texture details, we multiply the radiance hints with a neural encoding of the provisional synthesized image before passing it to DiLightNet. Finally, in the third stage, we resynthesize the background to be consistent with the lighting on the foreground object. We demonstrate and validate our lighting controlled diffusion model on a variety of text prompts and lighting conditions.
E4S: Fine-grained Face Swapping via Editing With Regional GAN Inversion
Tsinghua Shenzhen International Graduate School(清华大学深圳国际研究生院);South China University of Technology(华南理工大学);The University of Hong Kong(香港大学);Tencent AI Lab(腾讯AI实验室)
FiVA: Fine-grained Visual Attribute Dataset for Text-to-Image Diffusion Models
Stanford University(斯坦福大学);The Chinese University of Hong Kong(香港中文大学);Zhejiang University(浙江大学);S-Lab, NTU(南洋理工大学S-Lab实验室);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);CPII under InnoHK(InnoHK下属CPII机构)
RelightMaster: Precise Video Relighting with Multi-plane Light Images
Multimedia Laboratory, The Chinese University of Hong Kong(香港中文大学多媒体实验室);Kling Team, Kuaishou Technology(快手技术 Kling 团队);CPII under InnoHK(创新香港 CPII);Zhejiang University(浙江大学);Dalian University of Technology(大连理工大学)
AlbedoEdit: Unified Instance-Level Video Editing with Albedo Guidance
Max Planck Institute for Informatics(马克斯·普朗克信息研究所);University of California Santa Barbara(加州大学圣巴巴拉分校);NVIDIA Research(NVIDIA研究);Texas A & M University(德克萨斯A&M大学)
EditRefiner: A Human-Aligned Agentic Framework for Image Editing Refinement
Shanghai Jiao Tong University(上海交通大学);Vivo Mobile Communication Co., Ltd(Vivo移动通信有限公司);University of Electronic Science and Technology of China(电子科学与技术大学)
Training-Free Text-Guided Color Editing with Multi-Modal Diffusion Transformer
The Hong Kong University of Science and Technology(香港科学与技术大学);International Digital Economy Academy(国际数字经济学院);The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));Tsinghua University(清华大学);Astribot;StepFun
Decoupled Cross-Modal Alignment Network for Text-RGBT Person Retrieval and A High-Quality Benchmark
School of Computer Science and Technology, Anhui University(安徽大学计算机科学与技术学院);School of Artificial Intelligence, Anhui University(安徽大学人工智能学院);National Key Laboratory of Opto-Electronic Information Acquisition and Protection Technology(光电信息采集与防护技术国家重点实验室);Anhui Provincial Key Laboratory of Multimodal Cognitive Computation(安徽省多模态认知计算重点实验室)
A training-free framework for high-fidelity appearance transfer via diffusion transformers
School of Intelligence Science and Technology, Nanjing University(南京大学智能科学与技术学院);School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);School of New Media and Communication, Tianjin University(天津大学新媒体与传播学院);JIUTIAN Research(中移九天);State Key Laboratory of Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室)