arXiv 2406.127392024-06-18cs.CL

面向模型堆叠的自蒸馏解锁200+语言的跨语言自然语言理解

Self-Distillation for Model Stacking Unlocks Cross-Lingual NLU in 200+ Languages

Center For Artificial Intelligence and Data Science, University of Würzburg（维尔茨堡大学人工智能与数据科学中心）；IESEG School of Management（IESEG管理学院）；KU Leuven（鲁汶大学）；Language Technology Lab, University of Cambridge（剑桥大学语言技术实验室）

Fabian David Schmidt, Philipp Borchert, Ivan Vulić, Goran Glavaš

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对大语言模型低资源语言NLU能力不足的问题，研究通过样本高效自蒸馏将MT编码器集成到LLM骨干中，得到的MT-LLM兼具多语言对齐与丰富知识优势，在127种低资源语言的3项NLU任务上显著优于translate-test方法。

摘要：

大语言模型（LLM）不仅是文本生成的首选方案，也已成为自然语言理解（NLU）任务的常用解决方案。它们通过在网络规模语料上进行语言建模获取了广泛知识，在英语NLU任务上表现出色，但难以将其NLU能力拓展到代表性不足的语言。相比之下，机器翻译（MT）模型能生成优异的多语言表示，即便在低资源语言上也能实现强劲的翻译性能。然而，MT编码器缺乏LLM通过海量语料的语言建模训练所获得的、支撑全面NLU所需的知识。在本研究中，我们通过样本高效的自蒸馏将MT编码器直接集成到LLM骨干网络中，从而兼得两者优势。由此得到的MT-LLM保留了MT编码器固有的多语言表示对齐能力，让低资源语言能够利用以英语为中心的LLM中嵌入的丰富知识。我们将MT编码器与LLM融合在单个模型中，缓解了基于离散翻译的跨语言迁移（如translate-test方法）固有的翻译误差传播问题与MT解码的推理开销。在涵盖3项主流NLU任务、127种以低资源语言为主的评估中，MT-LLM在跨语言迁移上表现出极高的有效性。基于相同MT模型的情况下，MT-LLM大幅且持续优于translate-test方法，证明我们真正为LLM解锁了多语言自然语言理解能力。

英文摘要：

LLMs have become a go-to solution not just for text generation, but also for natural language understanding (NLU) tasks. Acquiring extensive knowledge through language modeling on web-scale corpora, they excel on English NLU, yet struggle to extend their NLU capabilities to underrepresented languages. In contrast, machine translation models (MT) produce excellent multilingual representations, resulting in strong translation performance even for low-resource languages. MT encoders, however, lack the knowledge necessary for comprehensive NLU that LLMs obtain through language modeling training on immense corpora. In this work, we get the best both worlds by integrating MT encoders directly into LLM backbones via sample-efficient self-distillation. The resulting MT-LLMs preserve the inherent multilingual representational alignment from the MT encoder, allowing lower-resource languages to tap into the rich knowledge embedded in English-centric LLMs. Merging the MT encoder and LLM in a single model, we mitigate the propagation of translation errors and inference overhead of MT decoding inherent to discrete translation-based cross-lingual transfer (e.g., translate-test). Evaluation spanning three prominent NLU tasks and 127 predominantly low-resource languages renders MT-LLMs highly effective in cross-lingual transfer. MT-LLMs substantially and consistently outperform translate-test based on the same MT model, showing that we truly unlock multilingual language understanding for LLMs.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。