Style2Talker：兼具情感风格与艺术风格的高分辨率说话头生成
Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style
AI总结：
针对现有说话头生成忽略情感与艺术风格的问题，本文提出Style2Talker方法，通过两阶段风格化结合自动标注的情感文本，实现高表现力的高分辨率说话头生成，效果优于现有最优方法。
摘要：
尽管音频驱动的说话头自动动画生成近年来受到越来越多的关注，但现有研究主要集中在实现唇部与音频的同步，忽略了生成富有表现力视频的两个关键要素：情感风格与艺术风格。本文提出了一种创新的音频驱动说话人脸生成方法Style2Talker，该方法包含两个风格化阶段，即Style-E（情感风格阶段）和Style-A（艺术风格阶段），将文本控制的情感风格与图像控制的艺术风格整合到最终输出中。为了解决视频对应的情感文本描述稀缺的问题，我们提出了一种无需人工标注的范式，利用大规模预训练模型自动为现有音视频数据集标注情感文本标签。结合合成的情感文本，Style-E阶段使用大规模CLIP模型提取情感表征，将其与音频信息结合，作为一个高效潜在扩散模型的条件，该模型用于生成3DMM模型的情感运动系数。在Style-A阶段，我们开发了一个系数驱动的运动生成器，以及嵌入在知名StyleGAN中的艺术风格专属路径，从而能够利用生成的情感运动系数和艺术风格源图像，合成高分辨率的艺术风格化说话头视频。此外，为了更好地保留图像细节并避免伪影，我们向StyleGAN输入从身份图像中提取的多尺度内容特征，并分别通过设计的内容编码器和细化网络对其中间特征图进行优化。大量实验结果表明，我们的方法在音唇同步效果、情感风格与艺术风格表现方面，均优于现有的最优方法。
英文摘要：
Although automatically animating audio-driven talking heads has recently received growing interest, previous efforts have mainly concentrated on achieving lip synchronization with the audio, neglecting two crucial elements for generating expressive videos: emotion style and art style. In this paper, we present an innovative audio-driven talking face generation method called Style2Talker. It involves two stylized stages, namely Style-E and Style-A, which integrate text-controlled emotion style and picture-controlled art style into the final output. In order to prepare the scarce emotional text descriptions corresponding to the videos, we propose a labor-free paradigm that employs large-scale pretrained models to automatically annotate emotional text labels for existing audiovisual datasets. Incorporating the synthetic emotion texts, the Style-E stage utilizes a large-scale CLIP model to extract emotion representations, which are combined with the audio, serving as the condition for an efficient latent diffusion model designed to produce emotional motion coefficients of a 3DMM model. Moving on to the Style-A stage, we develop a coefficient-driven motion generator and an art-specific style path embedded in the well-known StyleGAN. This allows us to synthesize high-resolution artistically stylized talking head videos using the generated emotional motion coefficients and an art style source picture. Moreover, to better preserve image details and avoid artifacts, we provide StyleGAN with the multi-scale content features extracted from the identity image and refine its intermediate feature maps by the designed content encoder and refinement network, respectively. Extensive experimental results demonstrate our method outperforms existing state-of-the-art methods in terms of audio-lip synchronization and performance of both emotion style and art style.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
