arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.07077cs.SDcs.AI
收藏

CALM:用于大音频-语言模型的类条件稀疏注意力向量

CALM: Class-Conditional Sparse Attention Vectors for Large Audio-Language Models

MIT CSAIL(麻省理工学院计算机科学与人工智能实验室);Tuebingen AI Center(图宾根人工智能中心);MIT-IBM Watson AI Lab(麻省理工-IBM沃森人工智能实验室)

Videet Mehta, Liming Wang, Hilde Kuehne, Rogerio Feris, James R. Glass, M. Jehanzeb Mirza

AI总结:

本文提出CALM,一种用于小样本分类的类条件稀疏注意力向量方法,通过学习类依赖的重要性权重提升音频和音频视觉分类性能,实验显示在多个基准上优于传统投票方法。

摘要:

大音频-语言模型(LALMs)在多种下游任务中表现出强大的零样本能力,如音频问答和抽象推理;然而,这些模型在某些判别任务(如音频分类)上仍落后于专用模型。最近的研究表明,LALM中的稀疏注意力头子集可通过简单的投票方案作为强大的判别特征提取器。然而,这些方法对所有选定的头分配均匀权重,隐含假设每个头在所有语义类别中贡献相等。在本文中,我们提出用于大音频-语言模型的类条件稀疏注意力向量,一种小样本分类方法,通过学习类依赖的重要性权重来优化注意力头。这种公式允许各个头专门处理不同的语义类别,并按其估计的可靠性比例参与集成预测。在多个小样本音频和音频视觉分类基准和任务上的实验表明,我们的方法在音频分类、音频-视觉分类和欺骗检测任务上分别比最先进的均匀投票方法高出14.52%、1.53%和8.35%的绝对增益。

英文摘要:

Large audio-language models (LALMs) exhibit strong zero-shot capabilities in multiple downstream tasks, such as audio question answering (AQA) and abstract reasoning; however, these models still lag behind specialized models for certain discriminative tasks (e.g., audio classification). Recent studies show that sparse subsets of attention heads within an LALM can serve as strong discriminative feature extractors for downstream tasks such as classification via simple voting schemes. However, these methods assign uniform weights to all selected heads, implicitly assuming that each head contributes equally across all semantic categories. In this work, we propose Class-Conditional Sparse Attention Vectors for Large Audio-Language Models, a few-shot classification method that learns class-dependent importance weights over attention heads. This formulation allows individual heads to specialize in distinct semantic categories and to contribute to ensemble predictions proportionally to their estimated reliability. Experiments on multiple few-shot audio and audiovisual classification benchmarks and tasks demonstrate that our method consistently outperforms state-of-the-art uniform voting-based approaches by up to 14.52%, 1.53%, 8.35% absolute gains for audio classification, audio-visual classification, and spoofing detection respectively.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑