多模态基础模型如何编码文本与语音？跨语言与跨模态表征分析
How do Multimodal Foundation Models Encode Text and Speech? An Analysis of Cross-Lingual and Cross-Modal Representations
AI总结：
本文分析三个多模态基础模型在跨语言文本与语音语义等价句子上的激活，揭示表征随层收敛、长度适配作用及模态与语言差距规律。
摘要：
多模态基础模型旨在创建一个统一的表征空间，使其脱离语言句法或模态差异等表层特征。为研究这一点，我们考察了三个近期模型的内部表征，分析模型在文本和语音模态下、跨语言语义等价句子的激活情况。研究发现：1）跨模态表征会随模型层加深而趋于一致，但专门处理文本和语音的初始层除外。2）长度适配对于缩小文本与语音之间的跨模态差距至关重要，不过当前方法的有效性主要局限于高资源语言。3）语音表现出比文本更大的跨语言差异。4）对于未明确训练以获得模态无关表征的模型，模态差距比语言差距更显著。
英文摘要：
Multimodal foundation models aim to create a unified representation space that abstracts away from surface features like language syntax or modality differences. To investigate this, we study the internal representations of three recent models, analyzing the model activations from semantically equivalent sentences across languages in the text and speech modalities. Our findings reveal that: 1) Cross-modal representations converge over model layers, except in the initial layers specialized at text and speech processing. 2) Length adaptation is crucial for reducing the cross-modal gap between text and speech, although current approaches' effectiveness is primarily limited to high-resource languages. 3) Speech exhibits larger cross-lingual differences than text. 4) For models not explicitly trained for modality-agnostic representations, the modality gap is more prominent than the language gap.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
