SAO-Instruct:使用自然语言指令进行自由形式音频编辑
SAO-Instruct: Free-form Audio Editing using Natural Language Instructions
AI总结:
本文提出SAO-Instruct,基于Stable Audio Open的模型,支持任意自然语言指令编辑音频,通过合成数据训练,在客观指标和主观听感上均优于现有方法。
摘要:
生成模型在从短文本描述合成高保真音频方面已取得显著进展。然而,使用自然语言编辑现有音频在很大程度上仍未得到充分探索。当前的方法要么需要编辑后音频的完整描述,要么受限于缺乏灵活性的预定义编辑指令。在这项工作中,我们提出了SAO-Instruct,一个基于Stable Audio Open的模型,能够使用任意自由形式的自然语言指令编辑音频片段。为了训练我们的模型,我们使用Prompt-to-Prompt、DDPM反演和手动编辑流程创建了一个音频编辑三元组(输入音频、编辑指令、输出音频)数据集。尽管部分基于合成数据训练,我们的模型能够很好地泛化到真实的野外音频片段和未见过的编辑指令。我们证明SAO-Instruct在客观指标上取得了有竞争力的性能,并在主观听感测试中优于其他音频编辑方法。为了鼓励未来的研究,我们发布了我们的代码和模型权重。
英文摘要:
Generative models have made significant progress in synthesizing high-fidelity audio from short textual descriptions. However, editing existing audio using natural language has remained largely underexplored. Current approaches either require the complete description of the edited audio or are constrained to predefined edit instructions that lack flexibility. In this work, we introduce SAO-Instruct, a model based on Stable Audio Open capable of editing audio clips using any free-form natural language instruction. To train our model, we create a dataset of audio editing triplets (input audio, edit instruction, output audio) using Prompt-to-Prompt, DDPM inversion, and a manual editing pipeline. Although partially trained on synthetic data, our model generalizes well to real in-the-wild audio clips and unseen edit instructions. We demonstrate that SAO-Instruct achieves competitive performance on objective metrics and outperforms other audio editing approaches in a subjective listening study. To encourage future research, we release our code and model weights.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。