解耦语言与文化以评估多语言大语言模型
Disentangling Language and Culture for Evaluating Multilingual Large Language Models
AI总结：
本文提出双重评估框架，通过解耦语言媒介与文化语境评估LLM的多语言能力，发现“文化-语言协同”现象并揭示其神经元激活机制，强调了跨文化语言评估的必要性。
摘要：
本文引入了一个双重评估框架，以全面评估LLM的多语言能力。通过沿语言媒介和文化语境维度分解评估，该框架能够细致分析LLM在母语和跨文化语境下跨语言处理问题的能力。对广泛的模型进行了广泛评估，揭示了一个显著的“文化-语言协同”现象：当问题在文化上与语言一致时，模型表现更好。通过可解释性探针进一步探索了这一现象，表明在语言的文化语境中，特定神经元的激活比例更高。该激活比例可作为模型训练期间评估多语言性能的潜在指标。我们的发现挑战了LLM（主要在英语数据上训练）在各语言间表现一致的普遍观点，并突出了文化和语言模型评估的必要性。代码见https://yingjiahao14.github.io/Dual-Evaluation/。
英文摘要：
This paper introduces a Dual Evaluation Framework to comprehensively assess the multilingual capabilities of LLMs. By decomposing the evaluation along the dimensions of linguistic medium and cultural context, this framework enables a nuanced analysis of LLMs' ability to process questions within both native and cross-cultural contexts cross-lingually. Extensive evaluations are conducted on a wide range of models, revealing a notable "CulturalLinguistic Synergy" phenomenon, where models exhibit better performance when questions are culturally aligned with the language. This phenomenon is further explored through interpretability probing, which shows that a higher proportion of specific neurons are activated in a language's cultural context. This activation proportion could serve as a potential indicator for evaluating multilingual performance during model training. Our findings challenge the prevailing notion that LLMs, primarily trained on English data, perform uniformly across languages and highlight the necessity of culturally and linguistically model evaluations. Our code can be found at https://yingjiahao14. github.io/Dual-Evaluation/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
