Cafe-Talk：通过多模态粗粒度与细粒度控制生成3D说话人脸动画
Cafe-Talk: Generating 3D Talking Face Animation with Multimodal Coarse- and Fine-grained Control
AI总结：
针对现有3D说话人脸方法细粒度表情控制不足的问题，提出多模态粗细粒度控制的扩散Transformer模型Cafe-Talk，通过两阶段训练、标签交换机制等实现解耦控制，在唇形同步与表情表现力上达SOTA。
摘要：
语音驱动的3D说话人脸方法需要同时具备精准的唇形同步能力与可控的表情生成能力。现有方法仅采用离散情感标签对整个序列的表情进行全局控制，限制了时空域内灵活的细粒度面部控制。我们提出了一种基于扩散Transformer的3D说话人脸生成模型Cafe-Talk，该模型同时融合粗粒度与细粒度多模态控制条件。然而，多条件的耦合问题难以实现令人满意的效果。为解耦语音音频与细粒度条件，我们采用两阶段训练流程：首先仅使用语音音频和粗粒度条件训练Cafe-Talk，随后通过提出的细粒度控制适配器逐步加入以动作单元（AUs）表示的细粒度指令，避免破坏语音-唇形同步效果。为解耦粗粒度与细粒度条件，我们设计了标签交换训练机制，确保细粒度条件的主导性。我们还提出了基于掩码的CFG技术，用于调节细粒度控制的触发时机与强度。此外，我们引入了带有文本-AU对齐功能的文本检测器，支持用户自然语言输入，进一步完善多模态控制能力。大量实验结果表明，Cafe-Talk在唇形同步和表情表现力上达到了当前最优水平，且在用户研究中其细粒度控制能力获得了广泛认可。项目主页：https://harryxd2018.github.io/cafe-talk/
英文摘要：
Speech-driven 3D talking face method should offer both accurate lip synchronization and controllable expressions. Previous methods solely adopt discrete emotion labels to globally control expressions throughout sequences while limiting flexible fine-grained facial control within the spatiotemporal domain. We propose a diffusion-transformer-based 3D talking face generation model, Cafe-Talk, which simultaneously incorporates coarse- and fine-grained multimodal control conditions. Nevertheless, the entanglement of multiple conditions challenges achieving satisfying performance. To disentangle speech audio and fine-grained conditions, we employ a two-stage training pipeline. Specifically, Cafe-Talk is initially trained using only speech audio and coarse-grained conditions. Then, a proposed fine-grained control adapter gradually adds fine-grained instructions represented by action units (AUs), preventing unfavorable speech-lip synchronization. To disentangle coarse- and fine-grained conditions, we design a swap-label training mechanism, which enables the dominance of the fine-grained conditions. We also devise a mask-based CFG technique to regulate the occurrence and intensity of fine-grained control. In addition, a text-based detector is introduced with text-AU alignment to enable natural language user input and further support multimodal control. Extensive experimental results prove that Cafe-Talk achieves state-of-the-art lip synchronization and expressiveness performance and receives wide acceptance in fine-grained control in user studies. Project page: https://harryxd2018.github.io/cafe-talk/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
