看清说话者：通过语音优先引导和区域细化生成高分辨率的说话面孔
See the Speaker: Crafting High-Resolution Talking Faces from Speech with Prior Guidance and Region Refinement
AI总结：
本研究提出一种通过语音直接生成高分辨率说话面孔的方法，结合扩散模型与统计先验，实现高质量视频生成。
摘要：
与现有方法依赖源图像作为外观参考并使用源语音生成运动不同，本工作提出了一种新的方法，直接从语音中提取信息，解决语音到说话面孔中的关键挑战。具体来说，我们首先采用语音到面孔肖像生成阶段，利用语音条件扩散模型结合统计面部先验和样本自适应加权模块，实现高质量肖像生成。在随后的语音驱动说话面孔生成阶段，我们将如嘴唇运动、面部表情和眼动等表现动态嵌入扩散模型的潜在空间，并进一步优化嘴唇同步使用区域增强模块。为了生成高分辨率输出，我们整合了预训练的基于Transformer的离散代码本与图像渲染网络，以端到端的方式增强视频帧细节。实验结果表明，我们的方法在HDTF、VoxCeleb和AVSpeech数据集上优于现有方法。值得注意的是，这是首个能够仅从单个语音输入生成高分辨率、高质量说话面孔视频的方法。
英文摘要：
Unlike existing methods that rely on source images as appearance references and use source speech to generate motion, this work proposes a novel approach that directly extracts information from the speech, addressing key challenges in speech-to-talking face. Specifically, we first employ a speech-to-face portrait generation stage, utilizing a speech-conditioned diffusion model combined with statistical facial prior and a sample-adaptive weighting module to achieve high-quality portrait generation. In the subsequent speech-driven talking face generation stage, we embed expressive dynamics such as lip movement, facial expressions, and eye movements into the latent space of the diffusion model and further optimize lip synchronization using a region-enhancement module. To generate high-resolution outputs, we integrate a pre-trained Transformer-based discrete codebook with an image rendering network, enhancing video frame details in an end-to-end manner. Experimental results demonstrate that our method outperforms existing approaches on the HDTF, VoxCeleb, and AVSpeech datasets. Notably, this is the first method capable of generating high-resolution, high-quality talking face videos exclusively from a single speech input.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
