多语言预训练与指令微调改善跨语言知识对齐，但仅停留在浅层
Multilingual Pretraining and Instruction Tuning Improve Cross-Lingual Knowledge Alignment, But Only Shallowly
AI总结：
本文提出CLiKA框架，从性能、一致性和传导性三个层面评估大语言模型的跨语言知识对齐，发现多语言预训练和指令微调虽有帮助但效果浅层，尤其无法显著提升知识传导性。
摘要：
尽管当前的大语言模型在英语知识检索方面表现出强大能力，但它们在不同语言上的能力并不均衡。为此，已有两种方法被提出以解决这一问题，即多语言预训练和多语言指令微调。然而，这些方法是否以及如何促进模型内部的跨语言知识对齐，目前尚不清楚。本文提出了CLiKA，一个系统性的框架，用于从性能（Performance）、一致性（Consistency）和传导性（Conductivity）三个层面评估大语言模型的跨语言知识对齐程度，并探究了多语言预训练和指令微调对对齐程度的影响。结果表明：虽然多语言预训练和指令微调都有利于跨语言知识对齐，但训练策略需要精心设计。具体而言，继续预训练以牺牲其他语言为代价改善目标语言的对齐，而混合预训练对其他语言的影响较小。此外，所有被测大语言模型的整体跨语言知识对齐，尤其是传导性层面，均不令人满意，且无论是多语言预训练还是指令微调都无法实质性地改善跨语言知识传导性。
英文摘要：
Despite their strong ability to retrieve knowledge in English, current large language models show imbalance abilities in different languages. Two approaches are proposed to address this, i.e., multilingual pretraining and multilingual instruction tuning. However, whether and how do such methods contribute to the cross-lingual knowledge alignment inside the models is unknown. In this paper, we propose CLiKA, a systematic framework to assess the cross-lingual knowledge alignment of LLMs in the Performance, Consistency and Conductivity levels, and explored the effect of multilingual pretraining and instruction tuning on the degree of alignment. Results show that: while both multilingual pretraining and instruction tuning are beneficial for cross-lingual knowledge alignment, the training strategy needs to be carefully designed. Namely, continued pretraining improves the alignment of the target language at the cost of other languages, while mixed pretraining affect other languages less. Also, the overall cross-lingual knowledge alignment, especially in the conductivity level, is unsatisfactory for all tested LLMs, and neither multilingual pretraining nor instruction tuning can substantially improve the cross-lingual knowledge conductivity.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
