BodyReLux: 时序一致的全身人体视频重照明
BodyReLux: Temporally Consistent Full-Body Video Relighting
AI总结：
本文提出BodyReLux，一种基于视频扩散的框架，用于在时序一致的方式下重照明全身人体表演。该方法利用混合数据集训练，结合传统静态单光源捕捉和新型动态表演捕捉技术，通过引入新的光照条件表示方法和数据增强管道，实现了高质量、鲁棒且时序一致的视频重照明。
摘要：
能够重照明人体表演是后期制作和内容创作中的基本任务。我们提出了BodyReLux，一种针对特定主体的视频扩散框架，用于在时序一致的方式下重照明全身人体表演。我们的模型是在一个混合的像素对齐视频重照明数据集上训练的，涵盖了多样化的光照条件、表演和视角组合。为了获得这样的数据集，我们结合了传统的静态单光源捕捉（OLAT）和一种新的动态表演捕捉方法，在其中两个平滑变化的光照序列被快速交错。由于光照操作在人类闪烁融合阈值之上，交错不会显得闪烁。我们从预训练的文本到视频模型中训练视频重照明模型，以充分利用生成先验来产生高质量视频。为了实现精确的光照控制，我们引入了一种新的光照条件方法，将每个光源表示为一个标记。我们进一步使用掩码注意力对光照序列进行条件处理，以支持动态光照控制。结合精心设计的数据增强管道，我们实现了高质量、鲁棒且时序一致的特定主体人体表演视频重照明。
英文摘要：
Being able to relight human performance is a fundamental task for post production and content creation. We present BodyReLux, a subject-specific video diffusion-based framework for relighting full-body human performances in a temporally consistent way. Our model is trained on a hybrid dataset of pixel-aligned video relighting pairs, covering a diverse combination of lighting conditions, performances and viewpoints. To acquire such dataset, we combine traditional static One-Light-at-a-Time (OLAT) capture and a novel dynamic performance capture in which two smoothly varying lighting sequences are rapidly interleaved. Because the lighting operates above the human flicker-fusion threshold, the interleaving does not appear to strobe. We train our video relighting model from a pretrained text-to-video model to fully leverage the generative priors for producing high quality videos. To achieve accurate lighting control, we introduce a new lighting conditioning method that represents each light source as a token. We further condition on sequences of lighting using masked attention to support dynamic lighting control. Together with a carefully designed data augmentation pipeline, we achieve photorealistic, robust, and temporally consistent video relighting of subject-specific human performances.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
