A fundamental characteristic of audio is its compositional nature. Audio-language models (ALMs) trained using a contrastive approach (e.g., CLAP) that learns a shared representation between audio and language modalities have improved performance in many downstream applications, including zero-shot audio classification, audio retrieval, etc. However, the ability of these models to effectively perform compositional reasoning remains largely unexplored and necessitates additional research. In this paper, we propose CompA, a collection of two expert-annotated benchmarks with a majority of real-world audio samples, to evaluate compositional reasoning in ALMs. Our proposed CompA-order evaluates how well an ALM understands the order or occurrence of acoustic events in audio, and CompA-attribute evaluates attribute-binding of acoustic events. An instance from either benchmark consists of two audio-caption pairs, where both audios have the same acoustic events but with different compositions. An ALM is evaluated on how well it matches the right audio to the right caption. Using this benchmark, we first show that current ALMs perform only marginally better than random chance, thereby struggling with compositional reasoning. Next, we propose CompA-CLAP, where we fine-tune CLAP using a novel learning method to improve its compositional reasoning abilities. To train CompA-CLAP, we first propose improvements to contrastive training with composition-aware hard negatives, allowing for more focused training. Next, we propose a novel modular contrastive loss that helps the model learn fine-grained compositional understanding and overcomes the acute scarcity of openly available compositional audios. CompA-CLAP significantly improves over all our baseline models on the CompA benchmark, indicating its superior compositional reasoning capabilities.
CoSTALA: Compositional Spatio-Temporal Audio-Language Alignment via Multi-Grain Hierarchical Contrastive Learning
Xi’an Jiaotong Liverpool University(西交利物浦大学);MiLM Plus, Xiaomi Inc.(小米公司MiLM Plus);University of Oulu(奥卢大学);Institute of Acoustics, Chinese Academy of Sciences(中国科学院声学研究所)
When Capabilities Fail to Compose: Diagnosing the Compositionality Gap in Large Audio-Language Models
National Taiwan University(国立台湾大学);NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE)(台湾大学人工智能卓越研究中心);ASUS Open Cloud Infrastructure Software Center(华硕开放云端基础设施软件中心)
CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction
National University of Singapore(新加坡国立大学);University of Science and Technology of China(中国科学技术大学);University of Cambridge(剑桥大学);University of Toronto(多伦多大学)
The Sonar Moment: Benchmarking Audio-Language Models in Audio Geo-Localization
The State Key Laboratory of Complex and Critical Software Environment, Beihang University(复杂与关键软件环境国家重点实验室,北航);Shanghai AI Laboratory(上海人工智能实验室);The Key Laboratory of Data Science and Intelligent Computing, International Innovation Institute, Beihang University(数据科学与智能计算重点实验室,国际创新研究院,北航)
WavJourney: Compositional Audio Creation with Large Language Models
University of Surrey(萨里大学);Queen Mary University of London(伦敦玛丽女王大学);Xi’an Jiaotong Liverpool University(西交利物浦大学);The Chinese University of Hong Kong(香港中文大学)
Deliberation Model for On-Device Spoken Language Understanding
Meta
我们提出了一种新的审议为基础的方法,端到端(E2 E)的口语语言理解(SLU),其中流自动语音识别(ASR)模型产生的第一遍假设和第二遍自然语言理解(NLU)组件生成的语义分析条件ASR的文本和音频嵌入。通过制定E2 E SLU作为一个广义的解码器,我们的系统能够支持复杂的组成语义结构。此外,ASR和NLU之间的参数共享使系统特别适合资源受限(设备上)环境;我们提出的方法在TOPv 2数据集(STOP)的口语版本上始终优于强管道NLU基线0.60%至0.65%。我们证明了文本和音频特征的融合,加上系统重写首过假设的能力,使我们的方法对ASR错误更加鲁棒。最后,我们表明,我们的方法可以显着减少从自然语音到合成语音训练时的退化,但需要更多的工作,使文本到语音(TTS)的一个可行的解决方案,扩大E2 E SLU。
ParaSpeechCLAP: A Dual-Encoder Speech-Text Model for Rich Stylistic Language-Audio Pretraining
Department of Computer Science, The University of Texas at Austin, USA(德克萨斯大学奥斯汀分校计算机科学系);Computer Science and Data Science, New York University, USA(纽约大学计算机科学与数据科学系)
CLEAR: A Dataset for Compositional Language and Elementary Acoustic Reasoning
NECOTIS;Sherbrooke University(舍布鲁克大学)
我们介绍了声学问答(AQA)在声学推理领域的任务。在这个任务中,智能体学习根据声学背景回答问题。为了促进这一领域的研究,我们提出了一种改编自CLEVR的数据生成范式(Johnson et al. 2017)。我们通过利用一组基本声音来生成声学场景。我们还提供了一些功能程序,可用于编写问题和答案,利用每个场景中的基本声音的属性之间的关系。我们提供各种大小的AQA数据集以及数据生成代码。作为一个初步的实验,以验证我们的数据,我们报告的准确性,当前国家的最先进的视觉问答模型时,他们被应用到AQA任务没有修改。虽然有很多基于文本、图像或视频数据的问答任务,但据我们所知,我们是第一个提出直接在音频流上回答问题的人。我们希望这一贡献将促进该领域研究的发展。
AudioStory: Generating Long-Form Narrative Audio with Large Language Models
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);ARC Lab, Tencent PCG(腾讯PCG ARC实验室);MAIS, Institute of Automation, CAS, Beijing(自动化研究所北京研究所MAIS)