ExpCLIP：通过语义对齐连接文本与面部表情
ExpCLIP: Bridging Text and Facial Expressions via Semantic Alignment
AI总结：
研究提出ExpCLIP模型，利用自然语言作为情绪提示控制语音驱动面部动画风格，通过构建TEAD数据集和EPA机制实现文本与表情的语义对齐，提升了动画表现力与风格控制灵活性。
摘要：
风格化语音驱动面部动画的目标是创建包含特定情绪表情的动画。现有方法通常依赖于预先设定的情绪标签或面部表情模板，这可能限制了准确传达用户意图所需的灵活性。在本研究中，我们引入一种技术，通过将自然语言作为情绪提示来实现对任意风格的控制。该技术在灵活性和用户友好性方面均具有优势。为实现此目标，我们首先构建了一个文本-表情对齐数据集（TEAD），其中每个面部表情都与多个提示式描述相配对。我们提出了一种由大型语言模型（LLM）支持的自动标注方法，以加速数据集构建，从而消除人工标注的高昂成本。随后，我们利用TEAD训练了一个基于CLIP的模型，称为ExpCLIP，该模型将文本和面部表情编码为语义对齐的风格嵌入。这些嵌入随后被集成到面部动画生成器中，以生成富有表现力且可控的面部动画。鉴于现有语音驱动面部动画训练数据中面部情绪的多样性有限，我们进一步引入了一种有效的表情提示增强（EPA）机制，使动画生成器能够在风格控制上支持前所未有的丰富度。综合实验表明，我们的方法实现了富有表现力的面部动画生成，并在有效传达所需风格方面提供了更高的灵活性。
英文摘要：
The objective of stylized speech-driven facial animation is to create animations that encapsulate specific emotional expressions. Existing methods often depend on pre-established emotional labels or facial expression templates, which may limit the necessary flexibility for accurately conveying user intent. In this research, we introduce a technique that enables the control of arbitrary styles by leveraging natural language as emotion prompts. This technique presents benefits in terms of both flexibility and user-friendliness. To realize this objective, we initially construct a Text-Expression Alignment Dataset (TEAD), wherein each facial expression is paired with several prompt-like descriptions.We propose an innovative automatic annotation method, supported by Large Language Models (LLMs), to expedite the dataset construction, thereby eliminating the substantial expense of manual annotation. Following this, we utilize TEAD to train a CLIP-based model, termed ExpCLIP, which encodes text and facial expressions into semantically aligned style embeddings. The embeddings are subsequently integrated into the facial animation generator to yield expressive and controllable facial animations. Given the limited diversity of facial emotions in existing speech-driven facial animation training data, we further introduce an effective Expression Prompt Augmentation (EPA) mechanism to enable the animation generator to support unprecedented richness in style control. Comprehensive experiments illustrate that our method accomplishes expressive facial animation generation and offers enhanced flexibility in effectively conveying the desired style.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
