通过翻译跟随示例增强指令调优大语言模型的跨语言能力
Empowering Cross-lingual Abilities of Instruction-tuned Large Language Models by Translation-following demonstrations
AI总结：
针对大语言模型跨语言能力不均衡问题，提出带翻译跟随示例的CrossAlpaca指令调优模型，通过语义对齐增强非英语能力，在多类基准六种语言上优于单语调优模型。
摘要：
大语言模型（LLMs）的语言能力往往因预训练数据分布不均衡而偏向英语。这种差异在后续微调中持续存在，影响着LLMs的跨语言能力。本文提出通过构建非英语语言与英语之间的语义对齐，增强指令调优大语言模型（It-LLMs）的非英语语言能力。为此，我们提出CrossAlpaca——一种具备跨语言指令跟随与翻译跟随示例的It-LLM，用于提升语言间的语义对齐效果。我们在多语言问答（QA）基准XQUAD、MLQA以及调整后的MMLU和BBH版本上验证了该方法。我们的模型在六种不同语言上的测试表现，优于基于单语数据调优的It-LLMs。最终结果表明，仅对非英语数据进行指令调优是不够的，翻译跟随示例可进一步提升语义对齐效果。
英文摘要：
The language ability of Large Language Models (LLMs) is often unbalanced towards English because of the imbalance in the distribution of the pre-training data. This disparity is demanded in further fine-tuning and affecting the cross-lingual abilities of LLMs. In this paper, we propose to empower Instructiontuned LLMs (It-LLMs) in languages other than English by building semantic alignment between them. Hence, we propose CrossAlpaca, an It-LLM with cross-lingual instruction-following and Translation-following demonstrations to improve semantic alignment between languages. We validate our approach on the multilingual Question Answering (QA) benchmarks XQUAD and MLQA and adapted versions of MMLU and BBH. Our models, tested over six different languages, outperform the It-LLMs tuned on monolingual data. The final results show that instruction tuning on non-English data is not enough and that semantic alignment can be further improved by Translation-following demonstrations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
