通过跨语言对齐增强多语言生成语言模型的上下文学习
Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignment
AI总结：
针对多语言生成模型偏向高资源语言且句子表示孤立的问题，提出利用翻译句对进行对比学习与指令对齐的框架，以极少 token 显著提升跨语言能力。
摘要：
多语言生成模型通过在大规模语料库上进行预训练，获得了显著的跨语言上下文学习能力。然而，它们仍然表现出对高资源语言的性能偏向，并学习到相互孤立的多语言句子表示分布，这可能会阻碍跨语言知识迁移。为弥合这一差距，我们提出了一个简单而有效的跨语言对齐框架，该框架利用翻译句子对。它通过多语言对比学习对齐不同语言之间的内部句子表示，并通过遵循目标语言中的跨语言指令来对齐输出。实验结果表明，即使使用不到预训练 token 数量 0.1‰ 的数据，我们的对齐框架也能显著提升生成语言模型的跨语言能力，并缩小性能差距。进一步分析表明，该框架使多语言模型获得了更优的内部多语言表示分布。
英文摘要：
Multilingual generative models obtain remarkable cross-lingual in-context learning capabilities through pre-training on large-scale corpora. However, they still exhibit a performance bias toward high-resource languages and learn isolated distributions of multilingual sentence representations, which may hinder knowledge transfer across languages. To bridge this gap, we propose a simple yet effective cross-lingual alignment framework exploiting pairs of translation sentences. It aligns the internal sentence representations across different languages via multilingual contrastive learning and aligns outputs by following cross-lingual instructions in the target language. Experimental results show that even with less than 0.1 {\textperthousand} of pre-training tokens, our alignment framework significantly boosts the cross-lingual abilities of generative language models and mitigates the performance gap. Further analyses reveal that it results in a better internal multilingual representation distribution of multilingual models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
