LumiSculpt：实现视频生成中一致的人像光照
LumiSculpt: Enabling Consistent Portrait Lighting in Video Generation
AI总结：
针对视频生成中光照与其他因素深度耦合、难以独立控制的问题，提出LumiSculpt模型，通过参考图像序列交互及可学习即插即用模块控制光源，构建LumiHuman数据集，实现精确一致的视频人像光照控制。
摘要：
光照在确保视频生成的自然度和美学质量方面起着关键作用。然而，光照的影响与视频的其他因素（如物体和场景）深度耦合。因此，独立解耦和建模连贯的光照条件仍然具有挑战性，这限制了在视频生成中控制光照的灵活性。本文受现有可控文本到图像（T2I）模型的启发，提出了LumiSculpt，它能在文本到视频（T2V）生成模型中实现精确且一致的光照控制。LumiSculpt为视频生成赋予了新的交互能力，允许输入具有自定义光照条件的参考图像序列。此外，LumiSculpt的核心可学习即插即用模块有助于直接控制视频扩散模型中假设光源的强度、位置和轨迹。为了有效训练LumiSculpt并解决光照数据不足的问题，我们构建了LumiHuman，这是一个新的轻量级且灵活的图像和视频人像光照数据集。实验结果表明，LumiSculpt在视频生成中实现了精确且高质量的光照控制。分析证明了LumiHuman的灵活性。
英文摘要：
Lighting plays a pivotal role in ensuring the naturalness and aesthetic quality of video generation. However, the impact of lighting is deeply coupled with other factors of videos, e.g., objects and scenes. Thus, it remains challenging to disentangle and model coherent lighting conditions independently, limiting the flexibility to control lighting in video generation. In this paper, inspired by the established controllable T2I models, we propose LumiSculpt, which enables precise and consistent lighting control in T2V generation models. LumiSculpt equips the video generation with new interactive capabilities, allowing the input of reference image sequences with customized lighting conditions. Furthermore, the core learnable plug-and-play module of LumiSculpt facilitates direct control over the intensity, position and trajectory of an assumed light source in video diffusion models. To effectively train LumiSculpt and address the issue of insufficient lighting data, we construct LumiHuman, a new lightweight and flexible dataset for portrait lighting of images and videos. Experimental results demonstrate that LumiSculpt achieves precise and high-quality lighting control in video generation. The analysis demonstrates the flexibility of LumiHuman.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
