arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.09527eess.AS
收藏

WavCraft:基于大型语言模型的音频编辑与生成

WavCraft: Audio Editing and Generation with Large Language Models

Jinhua Liang, Huan Zhang, Haohe Liu, Yin Cao, Qiuqiang Kong, Xubo Liu, Wenwu Wang, Mark D. Plumbley, Huy Phan, Emmanouil Benetos

AI总结:

本文提出 WavCraft 系统,利用 LLM 的上下文学习与任务分解能力调度多种音频模型,支持对话式音频编辑、生成及局部调整,实验显示其优于现有方法。

摘要:

我们介绍了 WavCraft,这是一个利用大型语言模型(LLM)连接多种任务专用模型,以进行音频内容创作和编辑的集合式系统。具体而言,WavCraft 用自然语言描述原始音频素材的内容,并根据音频描述和用户请求向 LLM 发出提示。WavCraft 利用 LLM 的上下文学习能力,将用户指令分解为若干任务,并由相应模块协作处理每一项任务。通过任务分解以及一组任务专用模型,WavCraft 遵循输入指令创建或编辑音频内容,使其具备更多细节和合理性,从而便于用户控制。此外,WavCraft 能够通过对话交互与用户协作,甚至在没有明确用户命令的情况下生成音频内容。实验表明,WavCraft 的性能优于现有方法,尤其是在调整音频片段的局部区域时。此外,WavCraft 能够遵循复杂指令,在输入录音的基础上编辑和创建音频内容,在更广泛的应用中为音频制作者提供便利。我们的实现和演示可在 https://github.com/JinhuaLiang/WavCraft 获取。

英文摘要:

We introduce WavCraft, a collective system that leverages large language models (LLMs) to connect diverse task-specific models for audio content creation and editing. Specifically, WavCraft describes the content of raw audio materials in natural language and prompts the LLM conditioned on audio descriptions and user requests. WavCraft leverages the in-context learning ability of the LLM to decomposes users' instructions into several tasks and tackle each task collaboratively with the particular module. Through task decomposition along with a set of task-specific models, WavCraft follows the input instruction to create or edit audio content with more details and rationales, facilitating user control. In addition, WavCraft is able to cooperate with users via dialogue interaction and even produce the audio content without explicit user commands. Experiments demonstrate that WavCraft yields a better performance than existing methods, especially when adjusting the local regions of audio clips. Moreover, WavCraft can follow complex instructions to edit and create audio content on the top of input recordings, facilitating audio producers in a broader range of applications. Our implementation and demos are available at this https://github.com/JinhuaLiang/WavCraft.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑