arXiv 2308.049482023-10-09cs.CL

通过语言对齐将大语言模型外推至非英语语言

Extrapolating Large Language Models to Non-English by Aligning Languages

Nanjing University（南京大学）；Peking University（北京大学）；Shanghai AI Lab（上海人工智能实验室）；The University of Hong Kong（香港大学）；University of California, Santa Barbara（加州大学圣巴巴拉分校）

Wenhao Zhu, Yunzhe Lv, Qingxiu Dong, Fei Yuan, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, Lei Li

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对大语言模型因训练数据不平衡导致的非英语能力较弱的问题，本文提出跨语言与多语言指令微调方法构建语义对齐，推导缩放定律优化数据分配，在多个基准上显著提升了非英语任务表现。

摘要：

由于训练数据的不平衡，现有大语言模型在不同语言上表现出的能力存在差异，其在英语任务上的表现通常优于其他语言的任务。本文通过构建跨语言语义对齐，提升预训练大语言模型在非英语语言上的能力。我们首先针对单种语言，在LLaMA（大语言模型元模型）上开展跨语言指令微调（CoIT，Cross-lingual Instruction-Tuning），即用翻译任务数据和跨语言通用任务数据对其进行微调，得到跨语言模型x-LLaMAs，并推导了底层缩放定律以探究使用可扩展翻译数据的优势。随后我们使用混合资源开展多语言指令微调（MuIT，Multilingual Instruction-Tuning），构建了多语言模型m-LLaMA。我们还说明了如何在资源受限的场景下利用缩放定律优化数据分配。在跨语言基准XQUAD和MLQA上的实验结果表明，在六种非英语语言中，x-LLaMAs的表现平均优于经英语指令微调的同类模型Alpaca 27.83%。在翻译数据集Flores-101上的评估结果显示，x-LLaMAs的表现平均优于此前基于LLaMA的模型18.89%。令人鼓舞的是，m-LLaMA在单种语言上的表现与x-LLaMAs相当，并且展现出遵循多语言指令的能力。对响应内容和表示空间的进一步分析，揭示了m-LLaMA中间层内多语言语义空间的对齐特性。

英文摘要：

Existing large language models show disparate capability across different languages, due to the imbalance in the training data. Their performances on English tasks are often stronger than on tasks of other languages. In this paper, we empower pre-trained LLMs on non-English languages by building semantic alignment across languages. We start from targeting individual languages by performing cross-lingual instruction-tuning (CoIT) on LLaMA, i.e. tuning it with translation task data and cross-lingual general task data to obtain cross-lingual models (x-LLaMAs), and formulate underlying scaling laws to investigate the advantages of using scalable translation data. Then we perform multilingual instruction-tuning (MuIT) with mixed resources to build multilingual m-LLaMA. We also illustrate how we leverage the scaling laws to optimize data allocation in a resource-constrained setting. Experiment results on cross-lingual benchmarks XQUAD and MLQA show that x-LLaMAs surpass the English instruction-tuned counterpart (Alpaca) by an average of 27.83% across six non-English languages. Evaluation results on translation dataset Flores-101 show that x-LLaMAs outperform previous LLaMA-based models by an average of 18.89%. Encouragingly, m-LLaMA achieves comparable performance to x-LLaMAs on individual languages and demonstrates the ability to follow multilingual instructions. Further analysis on response content and representation space reveals the alignment of the multilingual semantic space within the middle layers of m-LLaMA.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。