SonicDiffusion：基于预训练扩散模型的音频驱动图像生成与编辑
SonicDiffusion: Audio-Driven Image Generation and Editing with Pretrained Diffusion Models
AI总结：
SonicDiffusion提出将音频特征映射为token并注入预训练扩散模型，通过新增音频-图像交叉注意力层实现音频驱动的图像生成与编辑，实验表明性能优于现有方法。
摘要：
随着大规模文本到图像生成方法近期的成功，我们正在见证条件图像合成领域的一场革命。这一成功也为利用多模态输入控制生成和编辑过程开辟了新的机会。虽然利用深度、草图和其他图像等线索进行空间控制已经吸引了大量研究，但我们认为音频是另一种同样有效的模态，因为声音和视觉是人类感知的两个主要组成部分。因此，我们提出了一种在大规模图像扩散模型中实现音频条件控制的方法。我们的方法首先将从音频片段中获得的特征映射为token，这些token可以以类似于文本token的方式注入扩散模型。我们引入了额外的音频-图像交叉注意力层，并在冻结扩散模型原始层权重的同时对这些层进行微调。除了音频条件图像生成之外，我们的方法还可以与基于扩散的编辑方法结合使用，以实现音频条件图像编辑。我们在广泛的音频和图像数据集上展示了我们的方法。我们与近期方法进行了广泛的比较，并展示了良好的性能。
英文摘要：
We are witnessing a revolution in conditional image synthesis with the recent success of large scale text-to-image generation methods. This success also opens up new opportunities in controlling the generation and editing process using multi-modal input. While spatial control using cues such as depth, sketch, and other images has attracted a lot of research, we argue that another equally effective modality is audio since sound and sight are two main components of human perception. Hence, we propose a method to enable audio-conditioning in large scale image diffusion models. Our method first maps features obtained from audio clips to tokens that can be injected into the diffusion model in a fashion similar to text tokens. We introduce additional audio-image cross attention layers which we finetune while freezing the weights of the original layers of the diffusion model. In addition to audio conditioned image generation, our method can also be utilized in conjuction with diffusion based editing methods to enable audio conditioned image editing. We demonstrate our method on a wide range of audio and image datasets. We perform extensive comparisons with recent methods and show favorable performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
