语言融合用于参数高效的跨语言迁移
Language Fusion for Parameter-Efficient Cross-lingual Transfer
AI总结：
提出 FLARE 方法，在低秩适配器中融合源语言和目标语言表示，以参数高效方式提升跨语言迁移性能，在问答任务上较标准 LoRA 微调有显著提升。
摘要：
用于训练语言模型的多语言文本语料库的有限可用性，常常因除英语外的语言表示空间训练不足而导致下游任务性能不佳。这种“表示不足”促使最近的跨语言迁移方法利用英语表示空间，例如在输入层面混合英语和非英语标记，或扩展模型参数以适应新语言。然而，这些方法往往以增加计算复杂性为代价。我们提出了适配器中的语言表示融合（FLARE），一种新颖的方法，在保持参数效率的同时，提升除英语外语言的表示质量和下游性能。FLARE 在低秩（LoRA）适配器内通过轻量级线性变换整合源语言和目标语言表示，在保持参数效率的同时提高迁移性能。一系列跨语言自然语言理解代表性任务的实验，包括自然语言推理、问答和情感分析，证明了 FLARE 的有效性。与标准 LoRA 微调相比，FLARE 在问答任务上，以精确匹配指标衡量，对 Llama 3.1 实现了 4.9% 的性能提升，对 Gemma 2 实现了 2.2% 的性能提升。
英文摘要：
Limited availability of multilingual text corpora for training language models often leads to poor performance on downstream tasks due to undertrained representation spaces for languages other than English. This 'under-representation' has motivated recent cross-lingual transfer methods to leverage the English representation space by e.g. mixing English and 'non-English' tokens at the input level or extending model parameters to accommodate new languages. However, these approaches often come at the cost of increased computational complexity. We propose Fusion forLanguage Representations (FLARE) in adapters, a novel method that enhances representation quality and downstream performance for languages other than English while maintaining parameter efficiency. FLARE integrates source and target language representations within low-rank (LoRA) adapters using lightweight linear transformations, maintaining parameter efficiency while improving transfer performance. A series of experiments across representative cross-lingual natural language understanding tasks, including natural language inference, question-answering and sentiment analysis, demonstrate FLARE's effectiveness. FLARE achieves performance improvements of 4.9% for Llama 3.1 and 2.2% for Gemma~2 compared to standard LoRA fine-tuning on question-answering tasks, as measured by the exact match metric.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
