StyleAvatar：从单段视频生成实时照片级人像头像
StyleAvatar: Real-time Photo-realistic Portrait Avatar from a Single Video
AI总结：
针对人脸重演中2D GAN方法质量高但面部细粒度控制不足的问题，提出基于StyleGAN的StyleAvatar方法，通过组合表征、滑动窗口增强等实现实时高保真表情可控的人像头像生成，性能优于现有方法并落地了实时系统。
摘要：
人脸重演方法旨在尽可能逼真地还原并重新驱动人像视频。现有方法在质量与可控性之间存在两难困境：基于2D GAN的方法能实现更高的图像质量，但与3D方法相比，在面部属性的细粒度控制方面存在不足。在本研究中，我们提出StyleAvatar——一种基于StyleGAN网络的实时照片级人像头像重建方法，能够生成具备精准表情控制的高保真人像头像。我们通过引入组合表征和滑动窗口增强方法拓展了StyleGAN的能力，实现了更快的收敛速度并提升了泛化迁移能力。具体而言，我们将人像场景划分为三个部分以进行自适应调整：面部区域、非面部前景区域以及背景。此外，我们的网络结合了UNet、StyleGAN和时间编码的优势用于视频学习，从而实现高质量的视频生成。进一步地，我们提出了滑动窗口增强方法和预训练策略，分别用于提升泛化迁移能力和训练性能。所提出的网络可在两小时内收敛，同时保证高图像质量，且前向渲染时间仅为20毫秒。此外，我们构建了一个实时直播系统，进一步推动研究向应用落地。实验结果表明，与现有的人脸重演方法相比，我们的方法在图像质量、全身人像视频生成以及实时重驱动方面均具有优越性。本文的训练和推理代码可在https://github.com/LizhenWangT/StyleAvatar获取。
英文摘要：
Face reenactment methods attempt to restore and re-animate portrait videos as realistically as possible. Existing methods face a dilemma in quality versus controllability: 2D GAN-based methods achieve higher image quality but suffer in fine-grained control of facial attributes compared with 3D counterparts. In this work, we propose StyleAvatar, a real-time photo-realistic portrait avatar reconstruction method using StyleGAN-based networks, which can generate high-fidelity portrait avatars with faithful expression control. We expand the capabilities of StyleGAN by introducing a compositional representation and a sliding window augmentation method, which enable faster convergence and improve translation generalization. Specifically, we divide the portrait scenes into three parts for adaptive adjustments: facial region, non-facial foreground region, and the background. Besides, our network leverages the best of UNet, StyleGAN and time coding for video learning, which enables high-quality video generation. Furthermore, a sliding window augmentation method together with a pre-training strategy are proposed to improve translation generalization and training performance, respectively. The proposed network can converge within two hours while ensuring high image quality and a forward rendering time of only 20 milliseconds. Furthermore, we propose a real-time live system, which further pushes research into applications. Results and experiments demonstrate the superiority of our method in terms of image quality, full portrait video generation, and real-time re-animation compared to existing facial reenactment methods. Training and inference code for this paper are at https://github.com/LizhenWangT/StyleAvatar.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
