局部性对音频-语言模型中免训练音频令牌压缩的重要性
Locality Matters for Training-Free Audio Token Compression in Audio-Language Models
AI总结：
提出局部时间二分图合并（LTBM）方法，通过显式时间窗口约束合并相似邻近音频令牌，实现免训练的编码器空间压缩，并验证了局部性归纳偏置在音频令牌压缩中的任务依赖性优势。
摘要：
音频-语言模型（ALMs）越来越多地用于音频字幕生成、问答和开放式音频理解，但当音频输入表示为长前缀令牌序列时，其推理成本仍然很高。这些音频前缀消耗上下文预算，增加内存使用，并在资源受限或延迟敏感的环境中使部署更加困难。现有的免训练音频令牌缩减方法主要依赖于固定池化或基于分数的剪枝。固定池化是内容无关的，而基于分数的剪枝可以保留孤立的显著令牌但丢弃附近的声学上下文。我们提出局部时间二分图合并（LTBM），一种免训练的编码器空间压缩方法，在显式时间窗口约束下合并相似的邻近音频令牌。除了引入LTBM，我们还使用受控的全局合并变体来隔离时间局部性本身是否是音频令牌压缩的有用归纳偏置。在AudioCaps、Clotho和MMAU上使用Qwen2-Audio进行的实验显示了任务依赖的局部性效应：在几种压缩设置下，尤其是更强压缩下，局部感知合并更有利于字幕生成，而全局匹配在多项选择音频理解中更具竞争力。在Audio Flamingo 3上的跨骨干验证进一步支持了在适度和激进压缩下局部感知合并的字幕生成优势。
英文摘要：
Audio-language models (ALMs) are increasingly used for audio captioning, question answering, and open-ended audio understanding, but their inference cost remains high when audio inputs are represented as long prefix-token sequences. These audio prefixes consume context budget, increase memory usage, and make deployment harder in resource-constrained or latency-sensitive settings. Existing training-free audio-token reduction methods mainly rely on fixed pooling or score-based pruning. Fixed pooling is content-agnostic, while score-based pruning can preserve isolated salient tokens but discard nearby acoustic context. We propose Local Temporal Bipartite Merging (LTBM), a training-free encoder-space compression method that merges similar nearby audio tokens under an explicit temporal window constraint. Beyond introducing LTBM, we use a controlled Global Merge variant to isolate whether temporal locality itself is a useful inductive bias for audio-token compression. Experiments on AudioCaps, Clotho, and MMAU with Qwen2-Audio show evidence of a task-dependent locality effect: locality-aware merging is more favorable for captioning at several compression settings, especially under stronger compression, while global matching is more competitive for multiple-choice audio understanding. A cross-backbone validation on Audio Flamingo 3 further supports the captioning-side advantage of locality-aware merging under moderate and aggressive compression.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
