语音驱动的表现性说话嘴唇生成:基于条件序列生成对抗网络
Speech-Driven Expressive Talking Lips with Conditional Sequential Generative Adversarial Networks
AI总结:
本文提出条件序列生成对抗网络(CSG),利用语音信号直接提取的发音和情感特征生成逼真说话嘴唇运动,无需文本转录,在客观和主观评估中优于三个基线,并针对已知情感提出情感依赖模型改进效果。
摘要:
发音、情感和个性在口面部运动中起着重要作用。为了提高虚拟代理(VAs)的自然度和表现力,仔细建模这些因素之间的复杂相互作用至关重要。本文提出了一种条件生成对抗网络,称为条件序列生成对抗网络(CSG),它以原则性的方式学习情感与词汇内容之间的关系。该模型使用直接从语音信号中提取的一组发音和情感特征作为条件输入,生成逼真的运动。该方法的一个关键特点是它是一个语音驱动的框架,不需要文本转录。我们的实验在客观和主观评估方面显示了该模型相对于三个最先进基线的优越性。当目标情感已知时,我们提出通过用目标情感数据适配基础模型(CSG-Emo-Adapted)或将情感条件作为模型输入(CSG-Emo-Aware)来创建情感依赖模型。这些模型的客观评估显示,与CSG模型相比,CSG-Emo-Adapted的轨迹序列更接近原始序列,从而有所改进。主观评估显示,当目标情感为快乐时,该模型的结果显著优于CSG模型。
英文摘要:
Articulation, emotion, and personality play strong roles in the orofacial movements. To improve the naturalness and expressiveness of virtual agents (VAs), it is important that we carefully model the complex interplay between these factors. This paper proposes a conditional generative adversarial network, called conditional sequential GAN (CSG), which learns the relationship between emotion and lexical content in a principled manner. This model uses a set of articulatory and emotional features directly extracted from the speech signal as conditioning inputs, generating realistic movements. A key feature of the approach is that it is a speech-driven framework that does not require transcripts. Our experiments show the superiority of this model over three state-of-the-art baselines in terms of objective and subjective evaluations. When the target emotion is known, we propose to create emotionally dependent models by either adapting the base model with the target emotional data (CSG-Emo-Adapted), or adding emotional conditions as the input of the model (CSG-Emo-Aware). Objective evaluations of these models show improvements for the CSG-Emo-Adapted compared with the CSG model, as the trajectory sequences are closer to the original sequences. Subjective evaluations show significantly better results for this model compared with the CSG model when the target emotion is happiness.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。