arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.03335cs.SDcs.CVcs.LGeess.AS
收藏

Audio-Agent：利用大语言模型实现音频生成、编辑与合成

Audio-Agent: Leveraging LLMs For Audio Generation, Editing and Composition

HKUST（香港科技大学）；Dartmouth College（达特茅斯学院）

Zixuan Wang, Chi-Keung Tang, Yu-Wing Tai

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

该研究提出多模态框架Audio-Agent，结合GPT-4与预训练TTA扩散网络处理复杂文本条件的音频生成，通过微调Gemma2-2B-it实现视频转音频，为两类任务提供低训练开销的综合方案。

摘要：

我们提出了Audio-Agent，这是一个基于文本或视频输入实现音频生成、编辑与合成的多模态框架。传统的文本转音频（TTA）任务方法通常根据文本描述进行单次推理，这种设计虽然直接，但在面对复杂文本条件时难以生成高质量音频。在我们的方法中，我们采用预训练的TTA扩散网络作为音频生成智能体，与GPT-4协同工作：GPT-4将文本条件分解为原子化的具体指令，并调用智能体生成音频。通过这种方式，Audio-Agent能够生成与包含复杂多事件的文本或视频高度对齐的高质量音频，同时支持可变时长和可变音量的生成。对于视频转音频（VTA）任务，现有大多数方法需要训练时间戳检测器来同步视频事件与生成的音频，这一过程繁琐且耗时。我们则提出了一种更简单的方法：通过微调预训练大语言模型（LLM，例如Gemma2-2B-it）来获取语义和时间条件，以此搭建视频与音频模态之间的桥梁。因此，我们的框架为TTA和VTA任务提供了一套全面的解决方案，且不会带来过高的训练计算开销。

英文摘要：

We introduce Audio-Agent, a multimodal framework for audio generation, editing and composition based on text or video inputs. Conventional approaches for text-to-audio (TTA) tasks often make single-pass inferences from text descriptions. While straightforward, this design struggles to produce high-quality audio when given complex text conditions. In our method, we utilize a pre-trained TTA diffusion network as the audio generation agent to work in tandem with GPT-4, which decomposes the text condition into atomic, specific instructions and calls the agent for audio generation. In doing so, Audio-Agent can generate high-quality audio that is closely aligned with the provided text or video exhibiting complex and multiple events, while supporting variable-length and variable-volume generation. For video-to-audio (VTA) tasks, most existing methods require training a timestamp detector to synchronize video events with the generated audio, a process that can be tedious and time-consuming. Instead, we propose a simpler approach by fine-tuning a pre-trained Large Language Model (LLM), e.g., Gemma2-2B-it, to obtain both semantic and temporal conditions that bridge the video and audio modality. Consequently, our framework contributes a comprehensive solution for both TTA and VTA tasks without substantial computational overhead in training.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑