arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.05720cs.CV
收藏

HyperLips:用于说话人脸生成的高分辨率解码器超控制嘴唇

HyperLips: Hyper Control Lips with High Resolution Decoder for Talking Face Generation

University of Electronic Science and Technology of China(电子科技大学);ChengDu Sobey Digital Technology Co., Ltd(成都索贝数码科技股份有限公司);Peng Cheng Laboratory(鹏城实验室)

Yaosen Chen, Yu Yao, Zhiqiang Li, Wei Wang, Yanru Zhang, Han Yang, Xuming Wen

AI总结:

HyperLips提出两阶段框架,利用超网络控制嘴唇同步和高分辨率解码器提升人脸渲染质量,在说话人脸生成任务上超越现有方法。

摘要:

说话人脸生成在虚拟数字人领域具有广泛的应用前景。然而,现有的音频驱动说话人脸生成方法在确保嘴唇同步的同时渲染高保真面部视频仍然是一个挑战。为了解决这个问题,我们提出了HyperLips,一个两阶段框架,包括用于控制嘴唇的超网络和用于渲染高保真人脸的高分辨率解码器。在第一阶段,我们构建了一个基础人脸生成网络,该网络使用超网络来控制音频上视觉人脸信息的编码潜码。首先,使用FaceEncoder通过从包含人脸帧的视频源中提取视觉人脸信息的特征来获取潜码。然后,HyperConv(其加权参数由HyperNet以音频特征为输入进行更新)将修改潜码以使嘴唇运动与音频同步。最后,FaceDecoder将修改后的同步潜码解码为视觉人脸内容。在第二阶段,我们通过高分辨率解码器获得更高质量的人脸视频。为了进一步提高人脸生成质量,我们使用第一阶段生成的人脸图像和检测到的草图作为输入,训练了一个高分辨率解码器HRDecoder。大量的定量和定性实验表明,我们的方法在更真实、高保真和嘴唇同步方面优于最先进的工作。项目页面:https://semchan.github.io/HyperLips Project

英文摘要:

Talking face generation has a wide range of potential applications in the field of virtual digital humans. However, rendering high-fidelity facial video while ensuring lip synchronization is still a challenge for existing audio-driven talking face generation approaches. To address this issue, we propose HyperLips, a two-stage framework consisting of a hypernetwork for controlling lips and a high-resolution decoder for rendering high-fidelity faces. In the first stage, we construct a base face generation network that uses the hypernetwork to control the encoding latent code of the visual face information over audio. First, FaceEncoder is used to obtain latent code by extracting features from the visual face information taken from the video source containing the face frame.Then, HyperConv, which weighting parameters are updated by HyperNet with the audio features as input, will modify the latent code to synchronize the lip movement with the audio. Finally, FaceDecoder will decode the modified and synchronized latent code into visual face content. In the second stage, we obtain higher quality face videos through a high-resolution decoder. To further improve the quality of face generation, we trained a high-resolution decoder, HRDecoder, using face images and detected sketches generated from the first stage as input.Extensive quantitative and qualitative experiments show that our method outperforms state-of-the-art work with more realistic, high-fidelity, and lip synchronization. Project page: https://semchan.github.io/HyperLips Project/

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑