Lux Post Facto：基于条件视频扩散模型与混合数据集学习人像表演重打光
Lux Post Facto: Learning Portrait Performance Relighting with Conditional Video Diffusion and a Hybrid Dataset
AI总结：
本文提出Lux Post Facto，一种基于条件视频扩散模型和混合数据集的人像视频重打光方法，通过新的照明注入机制实现精确控制，在真实感和时间一致性上达到SOTA。
摘要：
视频人像重打光仍然具有挑战性，因为结果需要既具有真实感又在时间上稳定。这通常需要一个强大的模型设计来捕捉复杂的面部反射，以及在高质量的配对视频数据集（例如动态一次一光源（OLAT））上进行密集训练。在这项工作中，我们引入了Lux Post Facto，一种新颖的人像视频重打光方法，能够产生既具有真实感又在时间上一致的照明效果。从模型方面来看，我们设计了一个建立在最先进的预训练视频扩散模型之上的新型条件视频扩散模型，并辅以新的照明注入机制以实现精确控制。这样，我们利用强大的空间和时间生成能力，为不适定的重打光问题生成合理的解决方案。我们的技术使用由静态表情OLAT数据和野外人像表演视频组成的混合数据集，以联合学习重打光和时间建模。这避免了在不同照明条件下获取配对视频数据的需求。我们的大量实验表明，我们的模型在真实感和时间一致性方面都产生了最先进的结果。
英文摘要：
Video portrait relighting remains challenging because the results need to be both photorealistic and temporally stable. This typically requires a strong model design that can capture complex facial reflections as well as intensive training on a high-quality paired video dataset, such as dynamic one-light-at-a-time (OLAT). In this work, we introduce Lux Post Facto, a novel portrait video relighting method that produces both photorealistic and temporally consistent lighting effects. From the model side, we design a new conditional video diffusion model built upon state-of-the-art pre-trained video diffusion model, alongside a new lighting injection mechanism to enable precise control. This way we leverage strong spatial and temporal generative capability to generate plausible solutions to the ill-posed relighting problem. Our technique uses a hybrid dataset consisting of static expression OLAT data and in-the-wild portrait performance videos to jointly learn relighting and temporal modeling. This avoids the need to acquire paired video data in different lighting conditions. Our extensive experiments show that our model produces state-of-the-art results both in terms of photorealism and temporal consistency.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
