arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.14680eess.AScs.SD
收藏

基于扩散的文本到音乐生成:结合全局与局部文本条件

Diffusion based Text-to-Music Generation with Global and Local Text based Conditioning

Samsung R&D Institute UK (SRUK)(三星英国研发中心)

Jisi Zhang, Pablo Peso Parada, Md Asif Jalal, Karthikeyan Saravanan

AI总结:

本文提出一种基于扩散的文本到音乐生成模型,通过交叉注意力和FiLM机制分别融合T5局部文本表示与CLAP全局表示,并提出池化机制直接从T5提取全局嵌入以减少参数,实验证明该方法在生成质量与文本依从性间取得平衡且模型紧凑。

摘要:

基于扩散的文本到音乐(TTM)模型生成与文本描述对应的音乐。通常,基于UNet的扩散模型以预训练大语言模型或跨模态音频-语言表示模型生成的文本嵌入为条件。本工作提出一种基于扩散的TTM模型,其UNet同时以以下两者为条件:(i)通过交叉注意力接入的单模态语言模型(如T5),以及(ii)通过特征级线性调制(FiLM)接入的跨模态音频-语言表示模型(如CLAP)。该扩散模型被训练以利用来自T5的局部文本表示和来自CLAP的全局表示。此外,我们提出修改方案,通过我们称为平均池化和自注意力池化的机制从T5中提取全局和局部表示。该方法缓解了对额外编码器(如CLAP)提取全局表示的需求,从而减少了模型参数数量。结果表明,将CLAP全局嵌入加入T5局部嵌入可增强文本依从性(KL=1.47),对比仅依赖T5局部嵌入的基线模型(KL=1.54)。或者,通过提出的平均池化方法直接从T5局部嵌入中提取全局文本嵌入,可产生更优的生成质量(FAD=1.89),同时文本依从性(KL=1.51)略逊于以CLAP和T5文本嵌入同时为条件的模型(FAD=1.94和KL=1.47)。我们提出的解决方案不仅在效率上高效,而且在所需参数数量上更为紧凑。

英文摘要:

Diffusion based Text-To-Music (TTM) models generate music corresponding to text descriptions. Typically UNet based diffusion models condition on text embeddings generated from a pre-trained large language model or from a cross-modality audio-language representation model. This work proposes a diffusion based TTM, in which the UNet is conditioned on both (i) a uni-modal language model (e.g., T5) via cross-attention and (ii) a cross-modal audio-language representation model (e.g., CLAP) via Feature-wise Linear Modulation (FiLM). The diffusion model is trained to exploit both a local text representation from the T5 and a global representation from the CLAP. Furthermore, we propose modifications that extract both global and local representations from the T5 through pooling mechanisms that we call mean pooling and self-attention pooling. This approach mitigates the need for an additional encoder (e.g., CLAP) to extract a global representation, thereby reducing the number of model parameters. Our results show that incorporating the CLAP global embeddings to the T5 local embeddings enhances text adherence (KL=1.47) compared to a baseline model solely relying on the T5 local embeddings (KL=1.54). Alternatively, extracting global text embeddings directly from the T5 local embeddings through the proposed mean pooling approach yields superior generation quality (FAD=1.89) while exhibiting marginally inferior text adherence (KL=1.51) against the model conditioned on both CLAP and T5 text embeddings (FAD=1.94 and KL=1.47). Our proposed solution is not only efficient but also compact in terms of the number of parameters required.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑