MDT-A2G：探索用于协同语音手势生成的掩码扩散Transformer
MDT-A2G: Exploring Masked Diffusion Transformers for Co-Speech Gesture Generation
AI总结：
本文提出MDT-A2G，一种用于协同语音手势生成的掩码扩散Transformer，通过掩码建模强化手势序列时间关系，并融合音频、文本、情感和身份等多模态信息，同时利用高效推理策略实现显著加速，在保证生成质量的同时将学习和推理速度分别提升6倍和5.7倍以上。
摘要：
近年来，扩散Transformer领域的进展显著提升了高质量2D图像、3D视频和3D形状的生成效果。然而，Transformer架构在协同语音手势生成领域的有效性仍相对未被探索，因为先前的方法主要采用卷积神经网络（CNN）或仅使用少量简单的Transformer层。为了弥合这一研究空白，我们提出了一种新颖的用于协同语音手势生成的掩码扩散Transformer，称为MDT-A2G，它直接在手势序列上实现去噪过程。为了增强时间对齐的语音驱动手势的上下文推理能力，我们引入了一种新颖的掩码扩散Transformer。该模型采用了一种专门设计的掩码建模方案，以加强序列手势之间的时间关系学习，从而加快学习过程并产生连贯且逼真的动作。除了音频之外，我们的MDT-A2G模型还整合了多模态信息，包括文本、情感和身份。此外，我们提出了一种高效的推理策略，通过利用先前计算的结果来减少去噪计算量，从而在性能下降可忽略不计的情况下实现加速。实验结果表明，MDT-A2G在手势生成方面表现出色，其学习速度比传统扩散Transformer快6倍以上，推理速度是标准扩散模型的5.7倍。
英文摘要：
Recent advancements in the field of Diffusion Transformers have substantially improved the generation of high-quality 2D images, 3D videos, and 3D shapes. However, the effectiveness of the Transformer architecture in the domain of co-speech gesture generation remains relatively unexplored, as prior methodologies have predominantly employed the Convolutional Neural Network (CNNs) or simple a few transformer layers. In an attempt to bridge this research gap, we introduce a novel Masked Diffusion Transformer for co-speech gesture generation, referred to as MDT-A2G, which directly implements the denoising process on gesture sequences. To enhance the contextual reasoning capability of temporally aligned speech-driven gestures, we incorporate a novel Masked Diffusion Transformer. This model employs a mask modeling scheme specifically designed to strengthen temporal relation learning among sequence gestures, thereby expediting the learning process and leading to coherent and realistic motions. Apart from audio, Our MDT-A2G model also integrates multi-modal information, encompassing text, emotion, and identity. Furthermore, we propose an efficient inference strategy that diminishes the denoising computation by leveraging previously calculated results, thereby achieving a speedup with negligible performance degradation. Experimental results demonstrate that MDT-A2G excels in gesture generation, boasting a learning speed that is over 6$\times$ faster than traditional diffusion transformers and an inference speed that is 5.7$\times$ than the standard diffusion model.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
