arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2512.13495cs.CV
收藏

Soul:为高保真长期多模态动画注入生命

Soul: Breathe Life into Digital Human for High-fidelity Long-term Multimodal Animation

Tencent(腾讯)
Jiangning Zhang, Junwei Zhu, Zhenye Gan, Donghao Luo, Chuming Lin, Feifan Xu, Xu Peng, Jianlong Hu, Yuansen Liu, Yijia Hong, Weijian Cao, Han Feng, Xu Chen, Che… 展开作者

Jiangning Zhang, Junwei Zhu, Zhenye Gan, Donghao Luo, Chuming Lin, Feifan Xu, Xu Peng, Jianlong Hu, Yuansen Liu, Yijia Hong, Weijian Cao, Han Feng, Xu Chen, Chencan Fu, Keke He, Xiaobin Hu, Chengjie Wang

AI总结:

Soul通过多模态驱动框架实现高保真长期数字人类动画,结合自动化标注和优化推理效率,显著提升视频质量和唇同步精度。

摘要:

我们提出了一种多模态驱动的框架,称为Soul,用于高保真的长期数字人类动画,该框架能够从单张人物图像、文本提示和音频生成语义连贯的视频,实现精确的唇同步、生动的面部表情和稳定的身份数字身份。我们构建了Soul-1M,包含100万精细标注样本,其精确的自动化标注流程(涵盖人物、上半身、全身和多人场景)以缓解数据稀缺问题,同时我们精心编纂了Soul-Bench,用于全面公平地评估音频/文本引导的动画方法。该模型基于Wan2.2-5B主干网络,集成了音频注入层和多种训练策略,以及阈值感知的代码本替换,以确保长期生成的一致性。同时,步长/CFG蒸馏和轻量级VAE用于优化推理效率,实现了11.4倍的速度提升,质量损失可忽略不计。广泛实验表明,Soul在视频质量、视频-文本对齐、身份保持和唇同步准确性方面显著优于当前领先的开源和商用模型,证明了其在虚拟主播和影视制作等现实场景中的广泛应用前景。项目页面见https://zhangzjn.github.io/projects/Soul/

英文摘要:

We propose a multimodal-driven framework for high-fidelity long-term digital human animation termed $\textbf{Soul}$, which generates semantically coherent videos from a single-frame portrait image, text prompts, and audio, achieving precise lip synchronization, vivid facial expressions, and robust identity preservation. We construct Soul-1M, containing 1 million finely annotated samples with a precise automated annotation pipeline (covering portrait, upper-body, full-body, and multi-person scenes) to mitigate data scarcity, and we carefully curate Soul-Bench for comprehensive and fair evaluation of audio-/text-guided animation methods. The model is built on the Wan2.2-5B backbone, integrating audio-injection layers and multiple training strategies together with threshold-aware codebook replacement to ensure long-term generation consistency. Meanwhile, step/CFG distillation and a lightweight VAE are used to optimize inference efficiency, achieving an 11.4$\times$ speedup with negligible quality loss. Extensive experiments show that Soul significantly outperforms current leading open-source and commercial models on video quality, video-text alignment, identity preservation, and lip-synchronization accuracy, demonstrating broad applicability in real-world scenarios such as virtual anchors and film production. Project page at https://zhangzjn.github.io/projects/Soul/

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑