ReliTalk：从单段视频生成可重光照的说话人肖像
ReliTalk: Relightable Talking Portrait Generation from a Single Video
AI总结：
ReliTalk提出从单目视频生成可重光照的音频驱动说话人肖像，通过分解反射率和动态估计光照，解决跨场景适配问题，并在真实和合成数据集上验证了优越性。
摘要：
近年来，从单目视频创建生动的音频驱动肖像取得了巨大进展。然而，如何将生成的视频化身无缝适配到具有不同背景和光照条件的其他场景中，仍然是一个未解决的问题。另一方面，现有的重光照研究大多依赖动态光照或多视角数据，这些数据对于创建视频肖像而言过于昂贵。为弥合这一差距，我们提出了ReliTalk，一个用于从单目视频生成可重光照的音频驱动说话人肖像的新框架。我们的关键见解是从隐式学习的音频驱动面部法线和图像中分解肖像的反射率。具体而言，我们利用从音频特征中导出的三维面部先验，通过隐式函数预测精细的法线图。这些初始预测的法线随后通过动态估计给定视频的光照条件，在反射率分解中发挥关键作用。此外，我们使用在模拟多种光照条件下的身份一致性损失来优化立体面部表示，解决了单目视频中有限视角导致的病态问题。大量实验验证了我们提出的框架在真实和合成数据集上的优越性。我们的代码发布在https://github.com/arthur-qiu/ReliTalk。
英文摘要：
Recent years have witnessed great progress in creating vivid audio-driven portraits from monocular videos. However, how to seamlessly adapt the created video avatars to other scenarios with different backgrounds and lighting conditions remains unsolved. On the other hand, existing relighting studies mostly rely on dynamically lighted or multi-view data, which are too expensive for creating video portraits. To bridge this gap, we propose ReliTalk, a novel framework for relightable audio-driven talking portrait generation from monocular videos. Our key insight is to decompose the portrait's reflectance from implicitly learned audio-driven facial normals and images. Specifically, we involve 3D facial priors derived from audio features to predict delicate normal maps through implicit functions. These initially predicted normals then take a crucial part in reflectance decomposition by dynamically estimating the lighting condition of the given video. Moreover, the stereoscopic face representation is refined using the identity-consistent loss under simulated multiple lighting conditions, addressing the ill-posed problem caused by limited views available from a single monocular video. Extensive experiments validate the superiority of our proposed framework on both real and synthetic datasets. Our code is released in https://github.com/arthur-qiu/ReliTalk.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
