Soul:为高保真长期多模态动画注入生命
Soul: Breathe Life into Digital Human for High-fidelity Long-term Multimodal Animation
AI总结:
Soul通过多模态驱动框架实现高保真长期数字人类动画,结合自动化标注和优化推理效率,显著提升视频质量和唇同步精度。
摘要:
我们提出了一种多模态驱动的框架,称为Soul,用于高保真的长期数字人类动画,该框架能够从单张人物图像、文本提示和音频生成语义连贯的视频,实现精确的唇同步、生动的面部表情和稳定的身份数字身份。我们构建了Soul-1M,包含100万精细标注样本,其精确的自动化标注流程(涵盖人物、上半身、全身和多人场景)以缓解数据稀缺问题,同时我们精心编纂了Soul-Bench,用于全面公平地评估音频/文本引导的动画方法。该模型基于Wan2.2-5B主干网络,集成了音频注入层和多种训练策略,以及阈值感知的代码本替换,以确保长期生成的一致性。同时,步长/CFG蒸馏和轻量级VAE用于优化推理效率,实现了11.4倍的速度提升,质量损失可忽略不计。广泛实验表明,Soul在视频质量、视频-文本对齐、身份保持和唇同步准确性方面显著优于当前领先的开源和商用模型,证明了其在虚拟主播和影视制作等现实场景中的广泛应用前景。项目页面见https://zhangzjn.github.io/projects/Soul/
英文摘要:
We propose a multimodal-driven framework for high-fidelity long-term digital human animation termed $\textbf{Soul}$, which generates semantically coherent videos from a single-frame portrait image, text prompts, and audio, achieving precise lip synchronization, vivid facial expressions, and robust identity preservation. We construct Soul-1M, containing 1 million finely annotated samples with a precise automated annotation pipeline (covering portrait, upper-body, full-body, and multi-person scenes) to mitigate data scarcity, and we carefully curate Soul-Bench for comprehensive and fair evaluation of audio-/text-guided animation methods. The model is built on the Wan2.2-5B backbone, integrating audio-injection layers and multiple training strategies together with threshold-aware codebook replacement to ensure long-term generation consistency. Meanwhile, step/CFG distillation and a lightweight VAE are used to optimize inference efficiency, achieving an 11.4$\times$ speedup with negligible quality loss. Extensive experiments show that Soul significantly outperforms current leading open-source and commercial models on video quality, video-text alignment, identity preservation, and lip-synchronization accuracy, demonstrating broad applicability in real-world scenarios such as virtual anchors and film production. Project page at https://zhangzjn.github.io/projects/Soul/
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。