InstructAV2AV：基于指令的音频视频联合编辑
InstructAV2AV: Instruction-Guided Audio-Video Joint Editing
AI总结：
本文提出InstructAV2AV，首个端到端的指令引导音频视频联合编辑框架，通过构建大规模音频视频编辑数据集InsAVE-80K和改进的生成模型，实现了更高质量的音频视频联合编辑。
摘要：
最近的扩散基方法在视频内容操控方面取得了显著进展。然而，它们通常忽视伴随的音频，导致音频与编辑结果脱节。在本文中，我们提出了InstructAV2AV，首个端到端的指令引导音频视频联合编辑框架。我们首先开发了一个可扩展的数据合成管道，并构建了InsAVE-80K，首个大规模音频视频编辑数据集，包含高质量的源到目标配对。借助这一数据基础，我们适配了一个音频视频生成骨干网络，以利用其强大的先验知识。我们将音频视频输入与噪声潜在代码结合，以锚定源上下文，提出源指令门控注意力以提高指令遵循和内容保持，并引入两阶段训练策略以有效转移这些预训练的先验知识。广泛的实验表明，InstructAV2AV在两个评估集上，跨11个指标覆盖三个方面，均优于现有最先进方法，凸显了其在可控内容创作中的潜力。项目页面：https://hjzheng.net/projects/InstructAV2AV/.
英文摘要：
Recent diffusion-based methods have achieved impressive progress in video content manipulation. However, they typically ignore the accompanying audio, leaving the audio disjointed from the edited results. In this paper, we propose InstructAV2AV, the first end-to-end framework for instruction-guided audio-video joint editing. We first develop a scalable data synthesis pipeline and construct InsAVE-80K, the first large-scale audio-video editing dataset with high-quality source-to-target pairs. With this data foundation, we adapt an audio-video generation backbone to leverage its robust priors. We concatenate the audio-video input with noisy latent codes to anchor the source context, propose the source-instruction gated attention to improve instruction following and content preservation, and introduce a two-stage training strategy to effectively transfer these pre-trained priors. Extensive experiments demonstrate that InstructAV2AV outperforms state-of-the-art methods across 11 metrics spanning three aspects on two evaluation sets, highlighting its potential for controllable content creation. Project page: https://hjzheng.net/projects/InstructAV2AV/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
