AV-DiT：用于音视频联合生成的高效视听扩散 Transformer
AV-DiT: Efficient Audio-Visual Diffusion Transformer for Joint Audio and Video Generation
AI总结：
本文提出 AV-DiT，利用图像预训练的共享 DiT 主干与轻量级模态适配器实现音视频联合生成，并在 AIST++ 和 Landscape 上以更少可训练参数达到最先进性能。
摘要：
近期，扩散 Transformer（Diffusion Transformers，DiTs）在生成图像、视频和音频等高质量单模态内容方面展现出卓越能力。然而，基于 Transformer 的扩散器能否高效去除高斯噪声以实现出色的多模态内容创作，仍有待深入研究。为弥补这一空白，我们提出 AV-DiT，这是一种新颖且高效的视听扩散 Transformer，旨在生成同时包含视觉轨和音频轨的高质量、逼真视频。为最大限度降低模型复杂度和计算成本，AV-DiT 采用仅在图像数据上预训练的共享 DiT 主干，只有新插入的轻量级适配器可训练。该共享主干同时支持音频和视频生成。具体而言，视频分支在冻结的预训练 DiT 块中加入可训练的时间注意力层，以保证时间一致性。此外，少量可训练参数使基于图像的 DiT 块适配音频生成。一个额外的共享 DiT 块配备轻量级参数，用于促进音频与视觉模态之间的特征交互并确保对齐。在 AIST++ 和 Landscape 数据集上的大量实验表明，AV-DiT 以显著更少的可调参数，在联合视听生成任务中达到最先进性能。此外，结果表明，单一共享图像生成主干配合模态特定适配，便足以构建联合音视频生成器。源代码和预训练模型将公开发布。
英文摘要：
Recent Diffusion Transformers (DiTs) have shown impressive capabilities in generating high-quality single-modality content, including images, videos, and audio. However, it is still under-explored whether the transformer-based diffuser can efficiently denoise the Gaussian noises towards superb multimodal content creation. To bridge this gap, we introduce AV-DiT, a novel and efficient audio-visual diffusion transformer designed to generate high-quality, realistic videos with both visual and audio tracks. To minimize model complexity and computational costs, AV-DiT utilizes a shared DiT backbone pre-trained on image-only data, with only lightweight, newly inserted adapters being trainable. This shared backbone facilitates both audio and video generation. Specifically, the video branch incorporates a trainable temporal attention layer into a frozen pre-trained DiT block for temporal consistency. Additionally, a small number of trainable parameters adapt the image-based DiT block for audio generation. An extra shared DiT block, equipped with lightweight parameters, facilitates feature interaction between audio and visual modalities, ensuring alignment. Extensive experiments on the AIST++ and Landscape datasets demonstrate that AV-DiT achieves state-of-the-art performance in joint audio-visual generation with significantly fewer tunable parameters. Furthermore, our results highlight that a single shared image generative backbone with modality-specific adaptations is sufficient for constructing a joint audio-video generator. Our source code and pre-trained models will be released.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
