ALAS：音频语言模型的自动潜在对齐分数
ALAS: An Automatic Latent Alignment Score for Audio Language Models
AI总结：
提出ALAS指标，通过计算音频与文本表示的跨模态余弦相似度，无需训练即可评估语音-LLM的音频-文本对齐质量，揭示模型对齐深度与任务需求的关系。
摘要：
大型语言模型（LLM）被扩展为语音-LLM，它们学习的音频-文本对齐质量影响大多数下游口语理解（SLU）行为。然而，尽管融合策略不断增长，但没有标准方法来衡量语音-LLM内部如何将音频帧与文本标记绑定。我们引入ALAS（自动潜在对齐分数），一种模型和任务无关的度量，探测LLM的逐层隐藏状态，将音频和文本表示之间的跨模态余弦相似度与Whisper导出的参考进行评分。ALAS仅需要冻结的前向传递和现成的ASR参考，无需训练或拟合分类器，并校准到可解释的均匀基线，可在任务间比较。将ALAS应用于四个开源语音-LLM（AF3、Qwen2-Audio、Qwen-Omni、SALMONN），在情感识别（IEMOCAP）、开放式SQA（LibriSQA）和多选音频理解（MMAU-speech）上，我们发现对齐的深度和强度反映了每个模型的音频编码器设计以及任务的声学与语义需求，并且ALAS跟踪但不重复任务准确性，暴露了那些得分高但未真正基于音频的模型。我们将ALAS作为开源库发布，以便从业者探测自己的语音-LLM或在新任务上尝试。
英文摘要：
Large Language Models (LLMs) are extended into Speech-LLMs, and the quality of the audio--text alignment they learn affects most downstream Spoken Language Understanding (SLU) behavior. Yet despite a growth of fusion strategies, there is no standard way to measure how well a Speech-LLM internally binds audio frames to text tokens. We introduce ALAS (Automatic Latent Alignment Score), a model- and task-agnostic metric that probes the LLM's per-layer hidden states, scoring the cross-modal cosine similarity between audio and text representations against a Whisper-derived reference. ALAS needs only a frozen forward pass and an off-the-shelf ASR reference, with no training or fitted classifier, and is calibrated to an interpretable uniform baseline comparable across tasks. Applying ALAS to four open-source Speech-LLMs (AF3, Qwen2-Audio, Qwen-Omni, SALMONN) across emotion recognition (IEMOCAP), open-ended SQA (LibriSQA), and multi-choice audio understanding (MMAU-speech), we find that the depth and strength of alignment reflect each model's audio-encoder design and the acoustic-versus-semantic demands of the task, and that ALAS tracks but does not duplicate task accuracy, exposing models that score well without genuinely grounding in the audio. We release ALAS as an open-source library so that practitioners can probe their own Speech-LLMs or try new tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
