语言引导的联合音视频编辑：基于一次性适应方法
Language-Guided Joint Audio-Visual Editing via One-Shot Adaptation
AI总结：
本文提出语言引导的联合音视频编辑任务，采用一次性适应方法定制扩散模型，并引入跨模态语义增强以解决灾难性忽视，实验证明其有效性。
摘要：
在本文中，我们引入了一个新任务，称为语言引导的联合音视频编辑。给定一个发声事件的音频和图像对，该任务旨在通过基于语言引导编辑给定发声事件来生成新的音视频内容。例如，我们可以改变发声物体的背景环境，同时保持其外观不变，或者我们可以添加与视觉内容情境化的新声音。为了解决这一任务，我们提出了一种新的基于扩散模型的联合音视频编辑框架，并引入了两个关键思想。首先，我们提出了一种一次性适应方法，以定制生成扩散模型用于音视频内容编辑。仅需一个音视频样本，我们即可将音频和视觉扩散模型联合转移到目标领域。微调后，我们的模型能够实现该音视频样本的一致生成。其次，我们引入了一种跨模态语义增强方法。我们观察到，当使用语言作为内容编辑指导时，视觉分支可能会忽略编辑要求。这种现象被称为灾难性忽视，会阻碍内容编辑过程中的音视频对齐。因此，我们增强了语言与视觉之间的语义一致性以缓解此问题。大量实验验证了我们的方法在基于语言的音视频编辑中的有效性，并突出了其相对于几种基线方法的优越性。我们建议读者访问我们的项目页面以获取更多详细信息：https://liangsusan-git.github.io/project/avedit/。
英文摘要：
In this paper, we introduce a novel task called language-guided joint audio-visual editing. Given an audio and image pair of a sounding event, this task aims at generating new audio-visual content by editing the given sounding event conditioned on the language guidance. For instance, we can alter the background environment of a sounding object while keeping its appearance unchanged, or we can add new sounds contextualized to the visual content. To address this task, we propose a new diffusion-based framework for joint audio-visual editing and introduce two key ideas. Firstly, we propose a one-shot adaptation approach to tailor generative diffusion models for audio-visual content editing. With as few as one audio-visual sample, we jointly transfer the audio and vision diffusion models to the target domain. After fine-tuning, our model enables consistent generation of this audio-visual sample. Secondly, we introduce a cross-modal semantic enhancement approach. We observe that when using language as content editing guidance, the vision branch may overlook editing requirements. This phenomenon, termed catastrophic neglect, hampers audio-visual alignment during content editing. We therefore enhance semantic consistency between language and vision to mitigate this issue. Extensive experiments validate the effectiveness of our method in language-based audio-visual editing and highlight its superiority over several baseline approaches. We recommend that readers visit our project page for more details: https://liangsusan-git.github.io/project/avedit/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
