JambaTalk: 基于混合Transformer-Mamba模型的语音驱动3D说话人脸生成
JambaTalk: Speech-Driven 3D Talking Head Generation Based on Hybrid Transformer-Mamba Model
AI总结：
JambaTalk基于混合Transformer-Mamba模型，通过多模态整合实现语音驱动的3D说话人脸生成，实验表明其性能可与当前最先进模型相当。
摘要：
近年来，说话人脸生成已成为研究人员关注的焦点。大量努力致力于改进唇同步运动、捕捉表达性面部表情、生成自然头部姿态以及实现高质量视频。然而，尚未有单一模型在所有定量和定性指标上达到等效水平。我们引入Jamba，一种混合Transformer-Mamba模型，用于动画化3D人脸。Mamba，一种开创性的结构状态空间模型（SSM）架构，旨在克服传统Transformer架构的局限性，特别是在处理长序列方面。这一挑战限制了传统模型。Jamba结合了Transformer和Mamba方法的优势，提供了一种全面的解决方案。基于基础的Jamba块，我们提出JambaTalk，通过多模态整合增强运动多样性与唇同步。大量实验表明，我们的方法在性能上可与最先进模型相媲美。
英文摘要：
In recent years, the talking head generation has become a focal point for researchers. Considerable effort is being made to refine lip-sync motion, capture expressive facial expressions, generate natural head poses, and achieve high-quality video. However, no single model has yet achieved equivalence across all quantitative and qualitative metrics. We introduce Jamba, a hybrid Transformer-Mamba model, to animate a 3D face. Mamba, a pioneering Structured State Space Model (SSM) architecture, was developed to overcome the limitations of conventional Transformer architectures, particularly in handling long sequences. This challenge has constrained traditional models. Jamba combines the advantages of both the Transformer and Mamba approaches, offering a comprehensive solution. Based on the foundational Jamba block, we present JambaTalk to enhance motion variety and lip sync through multimodal integration. Extensive experiments reveal that our method achieves performance comparable or superior to state-of-the-art models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
