利用音频语言模型引导音频编辑
Guiding Audio Editing with Audio Language Model
AI总结：
SmartDJ框架利用音频语言模型推理与潜在扩散生成，将高级指令分解为原子操作，实现立体声音频编辑，显著提升感知质量、空间真实感与语义对齐。
摘要：
音频编辑在VR/AR沉浸体验、虚拟会议、声音设计及其他交互式媒体中扮演着核心角色。然而，最近的生成式音频编辑模型依赖于模板化的指令格式，并且仅限于单声道音频。这些模型无法处理声明式音频编辑，即用户声明期望的结果，而将编辑操作的细节留给系统。我们提出了SmartDJ，一个用于立体声音频编辑的新框架，它结合了音频语言模型的推理能力与潜在扩散的生成能力。给定一个高级指令，SmartDJ将其分解为一系列原子编辑操作，如添加、移除或空间重定位事件。然后，这些操作由一个训练用于操控立体声音频的扩散模型执行。为了支持这一点，我们设计了一个数据合成流水线，用于生成高级指令、原子编辑操作以及每次编辑操作前后音频的配对示例。实验表明，与先前的音频编辑方法相比，SmartDJ在感知质量、空间真实感和语义对齐方面均取得了更优的结果。演示可在https://zitonglan.github.io/project/smartdj/smartdj.html获取。
英文摘要：
Audio editing plays a central role in VR/AR immersion, virtual conferencing, sound design, and other interactive media. However, recent generative audio editing models depend on template-like instruction formats and are restricted to mono-channel audio. These models fail to deal with declarative audio editing, where the user declares what the desired outcome should be, while leaving the details of editing operations to the system. We introduce SmartDJ, a novel framework for stereo audio editing that combines the reasoning capability of audio language models with the generative power of latent diffusion. Given a high-level instruction, SmartDJ decomposes it into a sequence of atomic edit operations, such as adding, removing, or spatially relocating events. These operations are then executed by a diffusion model trained to manipulate stereo audio. To support this, we design a data synthesis pipeline that produces paired examples of high-level instructions, atomic edit operations, and audios before and after each edit operation. Experiments demonstrate that SmartDJ achieves superior perceptual quality, spatial realism, and semantic alignment compared to prior audio editing methods. Demos are available at https://zitonglan.github.io/project/smartdj/smartdj.html.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
