使用指令微调大语言模型与潜在扩散模型的文本到音频生成
Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Model
AI总结：
本文提出TANGO，利用指令微调LLM Flan-T5作为文本编码器并结合潜在扩散模型，在更小数据集上实现文本到音频生成性能超越AudioLDM。
摘要：
近期大规模语言模型（LLM）的巨大规模带来了许多有趣的特性，例如基于指令和思维链的微调，这显著提升了许多自然语言处理（NLP）任务中的零样本和少样本性能。受这些成功的启发，我们采用指令微调的LLM Flan-T5作为文本到音频（TTA）生成的文本编码器——该任务的目标是根据文本描述生成音频。先前的TTA工作要么预训练一个联合文本-音频编码器，要么使用非指令微调模型，如T5。因此，我们基于潜在扩散模型（LDM）的方法TANGO在AudioCaps测试集上的大多数指标上优于最先进的AudioLDM，并在其余指标上保持可比性，尽管我们在小63倍的数据集上训练LDM并保持文本编码器冻结。这一改进也可能归因于采用基于音频压力级的声音混合进行训练集增强，而先前方法采用随机混合。
英文摘要：
The immense scale of the recent large language models (LLM) allows many interesting properties, such as, instruction- and chain-of-thought-based fine-tuning, that has significantly improved zero- and few-shot performance in many natural language processing (NLP) tasks. Inspired by such successes, we adopt such an instruction-tuned LLM Flan-T5 as the text encoder for text-to-audio (TTA) generation -- a task where the goal is to generate an audio from its textual description. The prior works on TTA either pre-trained a joint text-audio encoder or used a non-instruction-tuned model, such as, T5. Consequently, our latent diffusion model (LDM)-based approach TANGO outperforms the state-of-the-art AudioLDM on most metrics and stays comparable on the rest on AudioCaps test set, despite training the LDM on a 63 times smaller dataset and keeping the text encoder frozen. This improvement might also be attributed to the adoption of audio pressure level-based sound mixing for training set augmentation, whereas the prior methods take a random mix.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
