语义感知的线性迁移：通过回收预训练语言模型实现跨语言迁移
Semantic Aware Linear Transfer by Recycling Pre-trained Language Models for Cross-lingual Transfer
AI总结：
本文提出语义感知线性迁移（SALT），通过回收目标语言预训练语言模型的嵌入，基于词汇重叠相似性推导回归线处理非重叠词嵌入，实现高效跨语言迁移，显著优于现有方法并加速收敛。
摘要：
大型语言模型（LLMs）越来越多地融入多语言能力，推动了将其迁移为特定目标语言模型的需求。然而，大多数方法通过用目标语言特有的词汇表替换源词汇表来混合源模型的嵌入，这可能会限制目标语言的表达能力，因为源模型主要是在英语数据上训练的。在本文中，我们提出了语义感知的线性迁移（SALT），一种新颖的跨语言迁移技术，它回收目标语言预训练语言模型（PLMs）的嵌入，将PLM派生的嵌入的深层表示优势传递给LLMs。SALT基于源词汇表和目标词汇表重叠部分的相似性推导出独特的回归线，以处理每个非重叠标记的嵌入空间。我们的大量实验表明，SALT显著优于其他迁移方法，并在语言适应过程中以更快的收敛速度实现更低的损失。值得注意的是，与其他方法相比，SALT在跨语言理解设置中取得了显著性能。此外，我们通过使用不同架构进行实验，强调了PLMs在增强当代LLMs功能方面的可扩展用途。
英文摘要：
Large Language Models (LLMs) increasingly incorporate multilingual capabilities, fueling the demand to transfer them into target language-specific models. However, most approaches, which blend the source model's embedding by replacing the source vocabulary with the target language-specific vocabulary, may constrain expressive capacity in the target language since the source model is predominantly trained on English data. In this paper, we propose Semantic Aware Linear Transfer (SALT), a novel cross-lingual transfer technique that recycles embeddings from target language Pre-trained Language Models (PLMs) to transmit the deep representational strengths of PLM-derived embedding to LLMs. SALT derives unique regression lines based on the similarity in the overlap of the source and target vocabularies, to handle each non-overlapping token's embedding space. Our extensive experiments show that SALT significantly outperforms other transfer methods and achieves lower loss with accelerating faster convergence during language adaptation. Notably, SALT obtains remarkable performance in cross-lingual understanding setups compared to other methods. Furthermore, we highlight the scalable use of PLMs to enhance the functionality of contemporary LLMs by conducting experiments with varying architectures.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
