FlowVQTalker：通过归一化流和量化生成高质量情感说话人脸
FlowVQTalker: High-Quality Emotional Talking Face Generation through Normalizing Flow and Quantization
AI总结：
本文提出FlowVQTalker，通过归一化流和向量量化建模生成高质量情感说话人脸，解决了音频到面部动态一对多映射及情感纹理细节缺失问题。
摘要：
生成情感说话人脸是一项实用但具有挑战性的任务。为了创建逼真的虚拟人，我们从人类视角汲取了两个关键见解：1）音频与非确定性面部动态（包括表情、眨眼、姿态）之间的联系应表现出同步和一对多的映射关系。2）生动的表情通常伴随着情感感知的高清（HD）纹理和精细的牙齿细节。然而，现有方法往往忽视了这两个方面。为此，本文提出使用归一化流和向量量化建模来生成同时满足上述两个见解的情感说话人脸（FlowVQTalker）。具体而言，我们开发了一个基于流的系数生成器，将面部情感动态编码为一个表示为混合分布的多情感类别潜在空间。生成过程从建模分布中进行随机采样开始，并在伴随音频的引导下，实现唇形同步和不确定的非语言面部线索生成。此外，我们设计的向量量化图像生成器将富有表现力的面部图像创建视为一个代码查询任务，利用学习到的码本提供丰富、高质量的纹理，以增强结果的情感感知。我们进行了大量实验以展示该方法的有效性。
英文摘要：
Generating emotional talking faces is a practical yet challenging endeavor. To create a lifelike avatar, we draw upon two critical insights from a human perspective: 1) The connection between audio and the non-deterministic facial dynamics, encompassing expressions, blinks, poses, should exhibit synchronous and one-to-many mapping. 2) Vibrant expressions are often accompanied by emotion-aware high-definition (HD) textures and finely detailed teeth. However, both aspects are frequently overlooked by existing methods. To this end, this paper proposes using normalizing Flow and Vector-Quantization modeling to produce emotional talking faces that satisfy both insights concurrently (FlowVQTalker). Specifically, we develop a flow-based coefficient generator that encodes the dynamics of facial emotion into a multi-emotion-class latent space represented as a mixture distribution. The generation process commences with random sampling from the modeled distribution, guided by the accompanying audio, enabling both lip-synchronization and the uncertain nonverbal facial cues generation. Furthermore, our designed vector-quantization image generator treats the creation of expressive facial images as a code query task, utilizing a learned codebook to provide rich, high-quality textures that enhance the emotional perception of the results. Extensive experiments are conducted to showcase the effectiveness of our approach.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
