arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.08753cs.SDcs.AIcs.CLeess.AS
收藏

CompA:解决音频语言模型中组合推理的差距

CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Models

University of Maryland, College Park(马里兰大学帕克分校);Adobe(奥多比公司);NVIDIA(英伟达公司)

Sreyan Ghosh, Ashish Seth, Sonal Kumar, Utkarsh Tyagi, Chandra Kiran Evuru, S. Ramaneswaran, S. Sakshi, Oriol Nieto, Ramani Duraiswami, Dinesh Manocha

AI总结:

本文提出 CompA 基准评估音频语言模型的组合推理能力,发现现有模型表现接近随机猜测,并通过组合感知困难负样本和模块化对比损失微调 CLAP 提出 CompA-CLAP,显著提升了组合推理性能。

摘要:

音频的一个基本特征是其组合性。使用对比方法(例如 CLAP)训练的音频语言模型(ALMs),通过学习音频和语言模态之间的共享表示,在许多下游应用中提升了性能,包括零样本音频分类、音频检索等。然而,这些模型有效执行组合推理的能力在很大程度上仍未被探索,需要进一步研究。在本文中,我们提出了 CompA,这是一个包含两个专家标注基准的集合,其中大部分为真实世界音频样本,用于评估 ALMs 中的组合推理。我们提出的 CompA-order 评估 ALM 理解音频中声学事件顺序或发生的能力,而 CompA-attribute 评估声学事件的属性绑定。任一基准中的一个实例由两个音频-文本对组成,其中两个音频具有相同的声学事件但组合不同。ALM 的评估基于其将正确音频匹配到正确文本的能力。使用此基准,我们首先表明当前的 ALMs 仅比随机猜测略好,从而在组合推理方面存在困难。接下来,我们提出 CompA-CLAP,其中我们使用一种新颖的学习方法微调 CLAP 以提高其组合推理能力。为了训练 CompA-CLAP,我们首先提出通过组合感知的困难负样本改进对比训练,从而实现更专注的训练。接着,我们提出一种新颖的模块化对比损失,帮助模型学习细粒度的组合理解,并克服公开可用组合音频的严重稀缺性。CompA-CLAP 在 CompA 基准上显著优于我们所有的基线模型,表明其卓越的组合推理能力。

英文摘要:

A fundamental characteristic of audio is its compositional nature. Audio-language models (ALMs) trained using a contrastive approach (e.g., CLAP) that learns a shared representation between audio and language modalities have improved performance in many downstream applications, including zero-shot audio classification, audio retrieval, etc. However, the ability of these models to effectively perform compositional reasoning remains largely unexplored and necessitates additional research. In this paper, we propose CompA, a collection of two expert-annotated benchmarks with a majority of real-world audio samples, to evaluate compositional reasoning in ALMs. Our proposed CompA-order evaluates how well an ALM understands the order or occurrence of acoustic events in audio, and CompA-attribute evaluates attribute-binding of acoustic events. An instance from either benchmark consists of two audio-caption pairs, where both audios have the same acoustic events but with different compositions. An ALM is evaluated on how well it matches the right audio to the right caption. Using this benchmark, we first show that current ALMs perform only marginally better than random chance, thereby struggling with compositional reasoning. Next, we propose CompA-CLAP, where we fine-tune CLAP using a novel learning method to improve its compositional reasoning abilities. To train CompA-CLAP, we first propose improvements to contrastive training with composition-aware hard negatives, allowing for more focused training. Next, we propose a novel modular contrastive loss that helps the model learn fine-grained compositional understanding and overcomes the acute scarcity of openly available compositional audios. CompA-CLAP significantly improves over all our baseline models on the CompA benchmark, indicating its superior compositional reasoning capabilities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑