arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.02569cs.CL
收藏

转写、翻译或音译:口语语言模型中中间表示的研究

Transcribe, Translate, or Transliterate: An Investigation of Intermediate Representations in Spoken Language Models

Stanford University(斯坦福大学);Toyota Technological Institute at Chicago(芝加哥丰田技术研究所)

Tolúlopé Ògúnrèmí, Christopher D. Manning, Dan Jurafsky, Karen Livescu

AI总结:

本研究通过分析三个口语语言模型的模态适配器输出,发现其表示策略取决于语音编码器训练任务:Whisper编码器采用英语中间语言表示语义,而其他编码器用英语单词表示语音特征。

摘要:

将语音与大型语言模型(LMs)集成的口语语言模型(SLMs)依赖模态适配器(MAs)将语音编码器的输出映射为解码器LM可理解的表示。然而,我们对这些关键的MAs如何转换表示知之甚少。在此,我们考察了三个SLM(SALMONN、Qwen2-Audio和Phi-4-Multimodal-Instruct)中MA的输出表示。通过找到与MA表示最接近的解码器LM标记,我们揭示了MA表示的两种策略。对于使用Whisper编码器的模型,MAs似乎使用基于英语的中间语言来表示输入的含义,使它们能够处理指令调优中未见过的语言。对于不这样做的模型,如Phi-4-Multimodal-Instruct,MAs则表示输入的语音特征,但用英语单词表达。我们假设,哪种策略出现取决于语音编码器是仅训练用于语音识别还是也用于翻译。

英文摘要:

Spoken language models (SLMs) that integrate speech with large language models (LMs) rely on modality adapters (MAs) to map the output of speech encoders to a representation that is understandable to the decoder LM. Yet we know very little about how these crucial MAs transform representations. Here we examine the MA output representation in three SLMs (SALMONN, Qwen2-Audio and Phi-4-Multimodal-Instruct). By finding the nearest decoder LM token to an MA representation, we uncover two strategies for MA representations. For models using a Whisper encoder, MAs appear to represent the meaning of the input using an English-based interlingua, allowing them to handle languages unseen in instruction tuning. For models that don't, like Phi-4-Multimodal-Instruct, MAs instead represent the phonetics of the input, but expressed with English words. We hypothesise that which arises depends on whether the speech encoder is trained only for speech recognition or also for translation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑