CoLLAP：基于音乐时间结构增强的对比式长音频-语言预训练
CoLLAP: Contrastive Long-form Language-Audio Pretraining with Musical Temporal Structure Augmentation
AI总结：
CoLLAP提出一种对比式长音频-语言预训练方法，通过音乐时间结构增强和注意力融合机制扩展音频与文本的感知窗口，并在长音频音乐文本检索及下游音乐信息检索任务中取得一致性能提升。
摘要：
对时间特征的建模在音频波形的表示学习中起着重要作用。我们提出对比式长音频-语言预训练（CoLLAP），以显著扩展输入音频（最长5分钟）和语言描述（超过250词）的感知窗口，同时支持跨模态和时间动态的对比学习。利用近期的Music-LLM为整首歌曲生成长篇音乐描述，并辅以音乐时间结构增强，我们从大规模AudioSet训练数据集中收集了51.3K个音频-文本对，其中平均音频时长达到288秒。我们提出一种新颖的对比学习架构，通过将每首歌曲分割为片段并提取其嵌入，将语言表示与结构化音频表示相融合。借助注意力机制，我们捕获多模态时间相关性，使模型能够自动加权并增强最终融合分数，从而改进对比对齐。最后，我们基于不同类型的骨干语言模型开发了两种CoLLAP模型变体。通过在多个长音频-音乐文本检索数据集上的全面实验，我们证明与基线相比，检索准确率得到了一致的性能提升。我们还展示了预训练的CoLLAP模型可以迁移到各种音乐信息检索任务中，并处理异构的长篇多模态上下文。
英文摘要：
Modeling temporal characteristics plays a significant role in the representation learning of audio waveform. We propose Contrastive Long-form Language-Audio Pretraining (\textbf{CoLLAP}) to significantly extend the perception window for both the input audio (up to 5 minutes) and the language descriptions (exceeding 250 words), while enabling contrastive learning across modalities and temporal dynamics. Leveraging recent Music-LLMs to generate long-form music captions for full-length songs, augmented with musical temporal structures, we collect 51.3K audio-text pairs derived from the large-scale AudioSet training dataset, where the average audio length reaches 288 seconds. We propose a novel contrastive learning architecture that fuses language representations with structured audio representations by segmenting each song into clips and extracting their embeddings. With an attention mechanism, we capture multimodal temporal correlations, allowing the model to automatically weigh and enhance the final fusion score for improved contrastive alignment. Finally, we develop two variants of the CoLLAP model with different types of backbone language models. Through comprehensive experiments on multiple long-form music-text retrieval datasets, we demonstrate consistent performance improvement in retrieval accuracy compared with baselines. We also show the pretrained CoLLAP models can be transferred to various music information retrieval tasks, with heterogeneous long-form multimodal contexts.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
