arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.04680cs.SDcs.AIcs.CLcs.MM
收藏

音频ControlNet用于细粒度音频生成与编辑

Audio ControlNet for Fine-Grained Audio Generation and Editing

Haina Zhu, Yao Xiao, Xiquan Li, Ziyang Ma, Jianwei Yu, Bowen Zhang, Mingqi Yang, Xie Chen

AI总结:

Audio ControlNet通过引入T2A-Adapter和T2A-Editor,实现了对音频响度、音调和事件的精细控制,并在AudioSet-Strong数据集上取得最佳性能。

摘要:

我们研究了细粒度文本到音频(T2A)生成任务。尽管最近的模型可以从文本描述中合成高质量音频,但它们通常缺乏对诸如响度、音调和声音事件等属性的精确控制。不同于先前方法重新训练模型以适应特定的控制类型,我们提出在预训练的T2A骨干网络之上训练ControlNet模型,以实现对响度、音调和事件的可控生成。我们引入了两种设计,T2A-ControlNet和T2A-Adapter,并展示了T2A-Adapter模型在结构上更加高效且具有强大的控制能力。仅需3800万额外参数,T2A-Adapter在AudioSet-Strong数据集上的事件级和片段级F1分数上均取得了最先进的性能。我们进一步将这一框架扩展到音频编辑领域,提出了T2A-Editor,用于根据指令指定的时间位置去除和插入音频事件。模型、代码、数据集流程和基准将被释放,以支持未来对可控音频生成和编辑的研究。

英文摘要:

We study the fine-grained text-to-audio (T2A) generation task. While recent models can synthesize high-quality audio from text descriptions, they often lack precise control over attributes such as loudness, pitch, and sound events. Unlike prior approaches that retrain models for specific control types, we propose to train ControlNet models on top of pre-trained T2A backbones to achieve controllable generation over loudness, pitch, and event roll. We introduce two designs, T2A-ControlNet and T2A-Adapter, and show that the T2A-Adapter model offers a more efficient structure with strong control ability. With only 38M additional parameters, T2A-Adapter achieves state-of-the-art performance on the AudioSet-Strong in both event-level and segment-level F1 scores. We further extend this framework to audio editing, proposing T2A-Editor for removing and inserting audio events at time locations specified by instructions. Models, code, dataset pipelines, and benchmarks will be released to support future research on controllable audio generation and editing.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑