HyperLips: Hyper Control Lips with High Resolution Decoder for Talking Face Generation
University of Electronic Science and Technology of China(电子科技大学);ChengDu Sobey Digital Technology Co., Ltd(成都索贝数码科技股份有限公司);Peng Cheng Laboratory(鹏城实验室)
Talking face generation has a wide range of potential applications in the field of virtual digital humans. However, rendering high-fidelity facial video while ensuring lip synchronization is still a challenge for existing audio-driven talking face generation approaches. To address this issue, we propose HyperLips, a two-stage framework consisting of a hypernetwork for controlling lips and a high-resolution decoder for rendering high-fidelity faces. In the first stage, we construct a base face generation network that uses the hypernetwork to control the encoding latent code of the visual face information over audio. First, FaceEncoder is used to obtain latent code by extracting features from the visual face information taken from the video source containing the face frame.Then, HyperConv, which weighting parameters are updated by HyperNet with the audio features as input, will modify the latent code to synchronize the lip movement with the audio. Finally, FaceDecoder will decode the modified and synchronized latent code into visual face content. In the second stage, we obtain higher quality face videos through a high-resolution decoder. To further improve the quality of face generation, we trained a high-resolution decoder, HRDecoder, using face images and detected sketches generated from the first stage as input.Extensive quantitative and qualitative experiments show that our method outperforms state-of-the-art work with more realistic, high-fidelity, and lip synchronization. Project page: https://semchan.github.io/HyperLips Project/
FACIAL: Synthesizing Dynamic Talking Face with Implicit Attribute Learning
The University of Texas at Dallas(德克萨斯大学达拉斯分校);Beihang University(北京航空航天大学);East China Normal University(华东师范大学);Xiamen University(厦门大学);Samsung Research America(三星美国研究院)
Assessing Identity Leakage in Talking Face Generation: Metrics and Evaluation Framework
Karlsruhe Institute of Technology(卡尔斯鲁厄理工学院);KIT Campus Transfer GmbH (KCT)(KIT校园转移公司);Istanbul Technical University(伊斯坦布尔技术大学);Carnegie Mellon University(卡内基梅隆大学)
Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expert
University of Electronic Science and Technology of China(电子科技大学);National University of Singapore(新加坡国立大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));University of Science and Technology Beijing(北京科技大学)