面向多语言基础模型的SeaEval：从跨语言对齐到文化推理
SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoning
AI总结：
本研究提出多语言基础模型基准SeaEval，从语义、多语言脆弱性及文化理解等维度测试开源与闭源模型，发现模型存在指令改写敏感、暴露偏差、多语言表现不一致等问题，为多语言多文化模型研究提供评估基础。
摘要：
我们提出了SeaEval——一个面向多语言基础模型的基准测试。除了衡量这些模型对自然语言的理解与推理能力外，我们还探究了它们对文化习俗、细微差异与价值观的理解程度。在标准准确率指标之外，我们还从语义和多语言两个维度研究基础模型的脆弱性。我们的分析覆盖开源与闭源模型，得出了经典NLP任务、推理及文化理解方面的实证结果。关键发现包括：（1）大多数模型在接收改写后的指令时会表现出不同行为。（2）许多模型仍存在暴露偏差问题（如位置偏差、多数标签偏差）。（3）对于基于事实、科学与常识知识的问题，语义等价的多语言查询理应得到一致的回答，但令人意外的是，大多数模型在这类查询上的表现并不一致。（4）经过多语言训练的模型尚未实现“均衡多语言”能力。我们的研究凸显了更具泛化性的语义表示与增强的多语言语境化能力的必要性。SeaEval可作为多语言、多文化场景下更深入研究与评估的启动平台。
英文摘要：
We present SeaEval, a benchmark for multilingual foundation models. In addition to characterizing how these models understand and reason with natural language, we also investigate how well they comprehend cultural practices, nuances, and values. Alongside standard accuracy metrics, we investigate the brittleness of foundation models in the dimensions of semantics and multilinguality. Our analyses span both open-sourced and closed models, leading to empirical results across classic NLP tasks, reasoning, and cultural comprehension. Key findings indicate (1) Most models exhibit varied behavior when given paraphrased instructions. (2) Many models still suffer from exposure bias (e.g., positional bias, majority label bias). (3) For questions rooted in factual, scientific, and commonsense knowledge, consistent responses are expected across multilingual queries that are semantically equivalent. Yet, most models surprisingly demonstrate inconsistent performance on these queries. (4) Multilingually-trained models have not attained "balanced multilingual" capabilities. Our endeavors underscore the need for more generalizable semantic representations and enhanced multilingual contextualization. SeaEval can serve as a launchpad for more thorough investigations and evaluations for multilingual and multicultural scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
