弥合差距：从单目手机捕捉创建影棚级虚拟化身
Bridging the Gap: Studio-like Avatar Creation from a Monocular Phone Capture
AI总结：
提出一种从单目手机捕捉生成影棚级纹理图的方法，利用 StyleGAN2 的 $W^+$ 空间参数化与微调，并结合图像梯度引导的扩散模型增强细节，生成照片级真实感且均匀照明的完整 3D 虚拟化身。
摘要：
为个人创建照片级真实感虚拟化身传统上需要使用复杂且昂贵的影棚设备（如 LightStage 系统）进行大量捕捉。尽管神经表示的最新进展使得通过快速手机扫描生成照片级真实感且可动画化的 3D 虚拟化身成为可能，但这些方法将捕捉时的光照烘焙在内，缺乏面部细节，并且在耳朵后部等区域存在缺失区域。因此，与影棚捕捉的虚拟化身相比，其质量仍有差距。在本文中，我们提出了一种弥合这一差距的方法，通过从短单目手机捕捉生成影棚级照明的纹理图来实现。我们通过使用 StyleGAN2 的 $W^+$ 空间对手机纹理图进行参数化，实现了近乎完美的重建。然后，我们利用一小组影棚捕捉的纹理作为对抗训练信号，在 $W^+$ 参数化空间中采样，对 StyleGAN2 进行微调。为了进一步提升面部细节的真实感和准确性，我们使用精心设计的扩散模型对 StyleGAN2 的输出进行超分辨率处理，该模型由手机捕捉纹理图的图像梯度引导。训练完成后，我们的方法在从日常单目智能手机视频生成影棚级面部纹理图方面表现出色。为展示其能力，我们展示了从单目手机捕捉生成照片级真实感、均匀照明且完整的虚拟化身。项目页面可在 http://shahrukhathar.github.io/2024/07/22/Bridging.html 找到。
英文摘要：
Creating photorealistic avatars for individuals traditionally involves extensive capture sessions with complex and expensive studio devices like the LightStage system. While recent strides in neural representations have enabled the generation of photorealistic and animatable 3D avatars from quick phone scans, they have the capture-time lighting baked-in, lack facial details and have missing regions in areas such as the back of the ears. Thus, they lag in quality compared to studio-captured avatars. In this paper, we propose a method that bridges this gap by generating studio-like illuminated texture maps from short, monocular phone captures. We do this by parameterizing the phone texture maps using the $W^+$ space of a StyleGAN2, enabling near-perfect reconstruction. Then, we finetune a StyleGAN2 by sampling in the $W^+$ parameterized space using a very small set of studio-captured textures as an adversarial training signal. To further enhance the realism and accuracy of facial details, we super-resolve the output of the StyleGAN2 using carefully designed diffusion model that is guided by image gradients of the phone-captured texture map. Once trained, our method excels at producing studio-like facial texture maps from casual monocular smartphone videos. Demonstrating its capabilities, we showcase the generation of photorealistic, uniformly lit, complete avatars from monocular phone captures. The project page can be found at http://shahrukhathar.github.io/2024/07/22/Bridging.html
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
