交错式语音语言模型在文本中隐式工作
Interleaved Speech Language Models Latently Work In Text
AI总结：
通过logit透镜分析发现，交错式语音-文本语言模型在中间层隐式转录语音为文本，然后基于文本空间预测下一个词，再转换回语音域，揭示了语音与文本模态的内部交互机制。
摘要：
语音语言模型（SLMs）已被广泛研究，常见范式是结合文本数据和预训练文本语言模型。一种领先的方法是语音-文本交错，即模型在包含语音和文本标记的序列上进行训练，旨在提升甚至仅语音能力。然而，这两种模态在模型潜在空间中的交互方式仍不清楚。在这项工作中，我们通过logit透镜的视角分析了来自不同模型家族和规模的交错式语音-文本语言模型，以提供这种洞察。我们揭示这些模型经历了一个隐式转录阶段，在该阶段中，口语单词的文本标记在中间层变得可解码，尽管模型并未针对语音识别进行训练。单词的转录出现在高达77%的数据的顶部候选词中。在此阶段之后，模型继续在文本空间中预测下一个词，然后再转换回语音域。最后，我们分析了交错数据和从文本语言模型初始化在引发这种行为中的作用，以及这如何与口语知识能力相关。我们的分析揭示了语音和文本模态之间关系的内部机制，并可能影响SLM的优化。
英文摘要：
Speech language models (SLMs) increasingly combine speech and text, often by interleaving their tokens within a single sequence. Yet how these two modalities interact in the model's latent space remains unclear. In this work, we analyze interleaved speech--text LMs from different model families and training configurations using three complementary methods. We reveal that these models pass through an implicit latent transcription phase in which the text token matching the spoken word becomes decodable in intermediate layers, despite not being trained for speech recognition. This phenomenon occurs in diverse, natural speech, and intermediate representations also encode likely text continuations. We further show that implicit transcription emerges most when combining text-LM pretraining and speech--text interleaving, and that its prevalence is positively associated with spoken factual-knowledge retrieval. Our analysis sheds light on the internal interaction between speech and text modalities in interleaved SLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
