LiveCLKTBench: 向多语言大语言模型中跨语言知识转移的可靠评估迈进
LiveCLKTBench: Towards Reliable Evaluation of Cross-Lingual Knowledge Transfer in Multilingual LLMs
AI总结：
本文提出LiveCLKTBench，通过自动化生成流程评估多语言大语言模型的跨语言知识转移，发现语言距离和方向对转移影响显著，且模型规模越大收益越小。
摘要：
评估大语言模型中的跨语言知识转移具有挑战性，因为目标语言中的正确答案可能源于真正的转移或预训练期间的先前暴露。我们提出了LiveCLKTBench，一个专门设计用于隔离和测量跨语言知识转移的自动化生成流程。我们的流程从现实领域中识别自包含的时间敏感知识实体，根据时间发生率进行过滤，并验证其与模型知识的一致性。这些有效实体的文档随后用于生成事实性问题，这些问题被翻译成多种语言以评估跨语言边界的知识转移性。使用LiveCLKTBench，我们评估了几个LLM在五种语言中的表现，发现跨语言转移受到语言距离的强烈影响，并且在不同语言方向上往往不对称。尽管较大的模型提高了转移能力，但随着规模的增加，收益逐渐减少，并且在不同领域中变化较大。这些发现为多语言转移提供了新的见解，并展示了LiveCLKTBench作为可靠基准在未来研究中的价值。
英文摘要：
Evaluating cross-lingual knowledge transfer in large language models is challenging, as correct answers in a target language may arise either from genuine transfer or from prior exposure during pre-training. We present LiveCLKTBench, an automated generation pipeline specifically designed to isolate and measure cross-lingual knowledge transfer. Our pipeline identifies self-contained, time-sensitive knowledge entities from real-world domains, filters them based on temporal occurrence, and verifies them against the model's knowledge. The documents of these valid entities are then used to generate factual questions, which are translated into multiple languages to evaluate transferability across linguistic boundaries. Using LiveCLKTBench, we evaluate several LLMs across five languages and observe that cross-lingual transfer is strongly influenced by linguistic distance and often asymmetric across language directions. While larger models improve transfer, the gains diminish with scale and vary across domains. These findings provide new insights into multilingual transfer and demonstrate the value of LiveCLKTBench as a reliable benchmark for future research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
