视觉与听觉：使用扩散潜变量对齐器的开放域视觉-音频生成
Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners
AI总结：
本文提出基于优化和 ImageBind 的多模态潜变量对齐器，在推理期引导扩散去噪，从而连接现成视频与音频模型，实现开放域联合及双向跨模态生成。
摘要：
视频和音频内容创作是电影产业及专业用户的核心技术。近来，现有基于扩散的方法分别处理视频与音频生成，这阻碍了该技术从学术界向产业界的转化。在本工作中，我们旨在填补这一空白，提出一个经过精心设计、基于优化的框架，用于跨视觉-音频以及视觉-音频联合生成。我们观察到现成视频或音频生成模型所具有的强大生成能力。因此，我们并非从头训练这些庞大的模型，而是提议通过一个共享的潜变量表示空间来连接现有强模型。具体而言，我们利用预训练的 ImageBind 模型提出一种多模态潜变量对齐器。我们的潜变量对齐器在核心上类似于分类器引导，后者会在推理阶段引导扩散去噪过程。通过精心设计的优化策略和损失函数，我们展示了本方法在视频-音频联合生成、视觉引导的音频生成以及音频引导的视觉生成任务上的优越性能。项目网站可参见 https://yzxing87.github.io/Seeing-and-Hearing/。
英文摘要：
Video and audio content creation serves as the core technique for the movie industry and professional users. Recently, existing diffusion-based methods tackle video and audio generation separately, which hinders the technique transfer from academia to industry. In this work, we aim at filling the gap, with a carefully designed optimization-based framework for cross-visual-audio and joint-visual-audio generation. We observe the powerful generation ability of off-the-shelf video or audio generation models. Thus, instead of training the giant models from scratch, we propose to bridge the existing strong models with a shared latent representation space. Specifically, we propose a multimodality latent aligner with the pre-trained ImageBind model. Our latent aligner shares a similar core as the classifier guidance that guides the diffusion denoising process during inference time. Through carefully designed optimization strategy and loss functions, we show the superior performance of our method on joint video-audio generation, visual-steered audio generation, and audio-steered visual generation tasks. The project website can be found at https://yzxing87.github.io/Seeing-and-Hearing/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
