基于音频引导的复杂多模态提示视觉编辑
Audio-Guided Visual Editing with Complex Multi-Modal Prompts
AI总结：
本文提出一种无需额外训练的音频引导视觉编辑框架，利用零样本多模态编码器、分离噪声分支和自适应块选择，有效融合文本与音频提示，解决复杂多模态编辑任务。
摘要：
基于扩散模型的视觉编辑已取得显著进展，但在复杂场景中，仅靠文本引导往往难以充分描述编辑需求，这凸显了引入额外非文本编辑提示的必要性。本文提出了一种新颖的音频引导视觉编辑框架，可在无需额外训练的情况下，利用多个文本和音频提示处理复杂编辑任务。现有的音频引导视觉编辑方法通常需要在特定数据集上训练，以使音频与文本对齐，这限制了其在真实场景中的泛化能力。我们利用具有强零样本能力的预训练多模态编码器，通过缓解音频编码器空间与扩散模型提示编码器空间之间的差异，将多样化的音频信息融入视觉编辑任务中。此外，我们提出了一种新颖的方法，通过分离噪声分支和自适应块选择来处理具有多个多模态编辑提示的复杂场景。我们在多种编辑任务上的综合实验表明，该框架通过融合丰富的音频信息，能够在仅靠文本方法失效的复杂编辑场景中表现出色。
英文摘要：
Visual editing with diffusion models has made significant progress but often struggles with complex scenarios that textual guidance alone could not adequately describe, highlighting the need for additional non-text editing prompts. In this work, we introduce a novel audio-guided visual editing framework that can handle complex editing tasks with multiple text and audio prompts without requiring additional training. Existing audio-guided visual editing methods often necessitate training on specific datasets to align audio with text, limiting their generalization to real-world situations. We leverage a pre-trained multi-modal encoder with strong zero-shot capabilities and integrate diverse audio into visual editing tasks, by alleviating the discrepancy between the audio encoder space and the diffusion model's prompt encoder space. Additionally, we propose a novel approach to handle complex scenarios with multiple and multi-modal editing prompts through our separate noise branching and adaptive patch selection. Our comprehensive experiments on diverse editing tasks demonstrate that our framework excels in handling complicated editing scenarios by incorporating rich information from audio, where text-only approaches fail.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
