FaceChain-ImagineID：从解耦音频自由生成高保真多样化说话人脸
FaceChain-ImagineID: Freely Crafting High-Fidelity Diverse Talking Faces from Disentangled Audio
AI总结：
本文提出FaceChain-ImagineID，通过渐进式音频解耦和可控连贯帧生成，在单一模型中从音频生成高保真、多样化且音频同步的说话人脸视频。
摘要：
在本文中，我们将人们听到语音、提取有意义线索并生成各种动态音频一致的说话人脸这一过程（称为“聆听与想象”）抽象为从单一音频生成高保真多样化说话人脸的任务。具体而言，该任务涉及两个关键挑战：一是从纠缠的音频中有效解耦身份、内容和情感，二是在视频内保持多样性、在视频间保持一致性。为解决这些问题，我们首先挖掘面部因素之间错综复杂的关系并简化解耦过程，专门设计了一种渐进式音频解耦方法，用于精确学习面部几何和语义，其中每个阶段都包含一个负责特定因素的定制训练模块。其次，为了在单一模型中仅从输入音频实现视觉多样且音频同步的动画，我们引入了可控连贯帧生成方法，该方法将三个可训练适配器与冻结的潜在扩散模型（LDMs）灵活集成，以专注于保持面部几何和语义，以及帧之间的纹理和时间连贯性。通过这种方式，我们继承了LDMs的高质量多样化生成能力，同时以较低的训练成本显著提高了其可控性。大量实验证明了我们的方法在处理这一范式时的灵活性和有效性。代码将在 https://github.com/modelscope/facechain 发布。
英文摘要：
In this paper, we abstract the process of people hearing speech, extracting meaningful cues, and creating various dynamically audio-consistent talking faces, termed Listening and Imagining, into the task of high-fidelity diverse talking faces generation from a single audio. Specifically, it involves two critical challenges: one is to effectively decouple identity, content, and emotion from entangled audio, and the other is to maintain intra-video diversity and inter-video consistency. To tackle the issues, we first dig out the intricate relationships among facial factors and simplify the decoupling process, tailoring a Progressive Audio Disentanglement for accurate facial geometry and semantics learning, where each stage incorporates a customized training module responsible for a specific factor. Secondly, to achieve visually diverse and audio-synchronized animation solely from input audio within a single model, we introduce the Controllable Coherent Frame generation, which involves the flexible integration of three trainable adapters with frozen Latent Diffusion Models (LDMs) to focus on maintaining facial geometry and semantics, as well as texture and temporal coherence between frames. In this way, we inherit high-quality diverse generation from LDMs while significantly improving their controllability at a low training cost. Extensive experiments demonstrate the flexibility and effectiveness of our method in handling this paradigm. The codes will be released at https://github.com/modelscope/facechain.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
