arXiv 2411.167482026-05-08cs.CV

具有内存库的多模态扩散变换器用于可扩展的长时谈话视频生成

Multimodal Diffusion Transformer with Memory Bank for Scalable Long-Duration Talking Video Generation

South China University of Technology（南方科技大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Beijing National Research Center for Information Science and Technology, Tsinghua University（北京信息科学研究中心，清华大学）；Department of Automation, BNRist, Tsinghua University（清华大学自动化系，北京信息科学研究中心）

Haojie Zhang, Zhihao Liang, Ruibo Fu, Bingyan Liu, Zhengqi Wen, Xuefei Liu, Jianhua Tao, Yaling Liang

备注：: 16 pages, 25 figures

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出 LetsTalk 框架，通过多模态指导和内存库机制解决长时谈话视频生成中的质量、一致性、时间连贯性和计算效率问题，实验表明其在生成质量和效率上达到新水平。

摘要：

长时谈话视频合成面临实现高质量视频、人物一致性、时间连贯性和计算效率的持续挑战。随着视频长度增加，视觉退化、人物漂移、时间伪影和误差累积等问题变得愈发严重，严重影响结果的现实性和可靠性。为了解决这些挑战，我们提出了 LetsTalk，一种配备多模态指导和新型内存库机制的扩散变换器框架，明确维护上下文连续性，并能够产生鲁棒、高质量且高效的长时谈话视频。特别地，LetsTalk 引入了噪声正则化的内存库以缓解在扩展视频生成过程中的误差累积和采样伪影。为进一步提高效率和时空建模， LetsTalk 采用了深度压缩自动编码器和具有线性注意力的时空感知变换器以实现有效的多模态融合。我们系统分析了三种融合方案，并表明结合深度（共生融合）用于人物特征和浅层（直接融合）用于音频可以实现更优的视觉真实性和精确的语音驱动运动，同时保持运动的多样性。广泛的实验表明， LetsTalk 在生成质量上建立了新的最先进水平，能够生成时间连贯且逼真的谈话视频，具有增强的多样性和活力，并且在参数上比先前方法少 8 倍，保持了显著的效率。

英文摘要：

Long-duration talking video synthesis faces enduring challenges in achieving high video quality, portrait consistency, temporal coherence, and computational efficiency. As video length increases, issues such as visual degradation, portrait drift, temporal artifacts, and error accumulation become increasingly problematic, severely affecting the realism and reliability of the results. To address these challenges, we present LetsTalk, a diffusion transformer framework equipped with multimodal guidance and a novel memory bank mechanism, explicitly maintaining contextual continuity and enabling robust, high-quality, and efficient generation of long-duration talking videos. In particular, LetsTalk introduces a noise-regularized memory bank to alleviate error accumulation and sampling artifacts during extended video generation. To further improve efficiency and spatiotemporal modeling, LetsTalk employs a deep compression autoencoder and a spatiotemporal-aware transformer with linear attention for effective multimodal fusion. We systematically analyze three fusion schemes and show that combining deep (Symbiotic Fusion) for portrait features and shallow (Direct Fusion) for audio achieves superior visual realism and precise speech-driven motion, while preserving diversity of movements. Extensive experiments demonstrate that LetsTalk establishes new state-of-the-art in generation quality, producing temporally coherent and realistic talking videos with enhanced diversity and liveliness, and maintains remarkable efficiency with 8x fewer parameters than previous approaches.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。