CPNet：利用基于CLIP的注意力凝聚器和概率图引导进行高保真说话人脸生成
CPNet: Exploiting CLIP-based Attention Condenser and Probability Map Guidance for High-fidelity Talking Face Generation
AI总结：
本文提出CPNet，通过基于CLIP的注意力凝聚器进行细粒度特征重校准，并利用面部关键点密度图作为辅助监督引导概率空间一致性，实现了高保真且高质量的唇形同步说话人脸生成。
摘要：
近年来，由于艰巨的挑战和广泛的应用场景（如电影动画和虚拟主播），说话人脸生成引起了计算机视觉研究界越来越多的关注。尽管已付出不懈努力来提高生成的说话人脸视频的保真度和唇形同步质量，但在合成质量和效率方面仍有很大的改进空间。实际上，这些尝试在某种程度上忽略了对细粒度特征提取/整合的探索以及关键点概率分布之间的一致性，从而导致局部细节模糊和保真度下降的问题反复出现。为了缓解这些困境，本文精心设计了一种新颖的基于CLIP的注意力与概率图引导网络（CPNet），用于推断高保真说话人脸视频。具体而言，考虑到细粒度特征重校准的需求，利用基于CLIP的注意力凝聚器，从流行的CLIP模型中转移具有丰富语义先验的知识。此外，为了保证概率空间的一致性并抑制关键点模糊性，我们创造性地提出将面部关键点密度图作为辅助监督信号，以引导生成帧的关键点分布学习。在广泛使用的基准数据集上的大量实验表明，我们的CPNet在图像和唇形同步质量方面优于现有技术。此外，还进行了一系列研究以消融各个关键组件的影响。
英文摘要：
Recently, talking face generation has drawn ever-increasing attention from the research community in computer vision due to its arduous challenges and widespread application scenarios, e.g. movie animation and virtual anchor. Although persevering efforts have been undertaken to enhance the fidelity and lip-sync quality of generated talking face videos, there is still large room for further improvements of synthesis quality and efficiency. Actually, these attempts somewhat ignore the explorations of fine-granularity feature extraction/integration and the consistency between probability distributions of landmarks, thereby recurring the issues of local details blurring and degraded fidelity. To mitigate these dilemmas, in this paper, a novel CLIP-based Attention and Probability Map Guided Network (CPNet) is delicately designed for inferring high-fidelity talking face videos. Specifically, considering the demands of fine-grained feature recalibration, a clip-based attention condenser is exploited to transfer knowledge with rich semantic priors from the prevailing CLIP model. Moreover, to guarantee the consistency in probability space and suppress the landmark ambiguity, we creatively propose the density map of facial landmark as auxiliary supervisory signal to guide the landmark distribution learning of generated frame. Extensive experiments on the widely-used benchmark dataset demonstrate the superiority of our CPNet against state of the arts in terms of image and lip-sync quality. In addition, a cohort of studies are also conducted to ablate the impacts of the individual pivotal components.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
