帧级内部工具使用用于音频语言模型中的时序定位
Encode Once, Decode Never: Reusing Audio LM Internals for Efficient Temporal Localization
AI总结：
本文提出帧级内部工具使用方法，通过二元分类器和IHP损失提升音频语言模型的时序定位能力，实现50倍加速和鲁棒长度泛化。
摘要：
大型音频语言模型越来越多地用于复杂的音频理解任务，但它们在需要精确时序定位的任务上表现不佳，例如词对齐和说话人识别。标准方法，即通过文本标记序列生成时间戳，计算成本高且容易产生幻觉，尤其是在处理超出模型训练分布的音频长度时。在本工作中，我们提出了一种帧级内部工具使用方法，训练音频语言模型直接使用其自身的内部音频表示进行时序定位。我们引入了一种轻量级的预测机制，通过两种目标进行训练：二元帧分类器和一种新的非均匀泊松过程（IHP）损失，该损失建模时间事件强度。在词定位、说话人识别和事件定位任务中，我们的方法优于基于标记的基线。最值得注意的是，它实现了超过50倍的推理加速，并展示了鲁棒的长度泛化能力，在标准标记基线完全崩溃的超出分布音频持续时间内仍能保持高精度。
英文摘要：
Audio language models process input audio into rich frame-level representations, but the standard approach to temporal localization generates timestamps as sequences of text tokens, which discards the frame-level representations in favor of autoregressive decoding. However, generating timestamps as tokens is slow and not parallelizable, and tends to hallucinate when producing timestamps outside the training distribution. We propose internal frame-level reuse, a method that trains audio LMs to reuse their own internal audio representations for temporal localization directly, bypassing token generation altogether. We introduce a lightweight prediction head trained via different frame-level objectives: a binary frame classifier and a novel inhomogeneous Poisson process (IHP) loss that models temporal event intensity. Across word localization, speaker diarization, and event localization tasks, our approach can achieve a >50x inference speedup over token-based generation and demonstrates robust length generalization, maintaining high accuracy on out-of-distribution audio durations where token-based models collapse completely. We find that reusing audio frame-level representations yields localization accuracies comparable to (and often better than) finetuned token-based baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
