arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.07919eess.AS
收藏

面向对比式语言-音频预训练的多粒度对齐改进研究

Advancing Multi-grained Alignment for Contrastive Language-Audio Pre-training

Yiming Li, Zhifang Guo, Xiangdong Wang, Hong Liu

AI总结:

针对现有音频-语言对比预训练模型忽略帧级文本对应、细粒度对齐不足的问题,提出基于共享码本、局部感知模块与难负例损失的多粒度对齐方法,在11项零样本任务上显著超越CLAP基线,性能比肩或优于当前最优水平。

摘要:

音频-语言联合学习领域已取得诸多进展,例如CLAP(对比式语言-音频预训练)模型就在多模态理解任务中展现出优异成效。这类模型通常将单模态局部表征(即帧特征或词特征)聚合为全局表征,再基于全局表征施加对比损失以实现粗粒度跨模态对齐。然而,帧级别与文本的对应关系可能被忽略,导致模型在可解释性和细粒度任务上存在缺陷,甚至会削弱粗粒度任务的性能。本研究旨在在大规模对比预训练中同时提升粗粒度和细粒度的音频-语言对齐效果。为统一两种模态的粒度与潜在分布,研究采用共享码本以公共基元表示多模态全局特征,并对每个码字施加正则化以编码模态共享语义,弥合帧特征与词特征之间的差距。在此基础上,引入局部感知模块提纯局部模式,并设计难负例引导损失函数强化对齐效果。在11项零样本粗粒度与细粒度任务上的实验表明,该模型不仅显著优于基线模型CLAP,与当前SOTA(最优水平)工作相比也能取得更优或具有竞争力的结果。

英文摘要:

Recent advances have been witnessed in audio-language joint learning, such as CLAP, that shows much success in multi-modal understanding tasks. These models usually aggregate uni-modal local representations, namely frame or word features, into global ones, on which the contrastive loss is employed to reach coarse-grained cross-modal alignment. However, frame-level correspondence with texts may be ignored, making it ill-posed on explainability and fine-grained challenges which may also undermine performances on coarse-grained tasks. In this work, we aim to improve both coarse- and fine-grained audio-language alignment in large-scale contrastive pre-training. To unify the granularity and latent distribution of two modalities, a shared codebook is adopted to represent multi-modal global features with common bases, and each codeword is regularized to encode modality-shared semantics, bridging the gap between frame and word features. Based on it, a locality-aware block is involved to purify local patterns, and a hard-negative guided loss is devised to boost alignment. Experiments on eleven zero-shot coarse- and fine-grained tasks suggest that our model not only surpasses the baseline CLAP significantly but also yields superior or competitive results compared to current SOTA works.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑