arXiv 2604.192622026-04-21cs.CLcs.AI

CulturALL：基于 grounded 任务的 LLM 多语言和多文化能力基准测试

CulturALL: Benchmarking Multilingual and Multicultural Competence of LLMs on Grounded Tasks

Alibaba Group（阿里巴巴集团）；Beijing Language and Culture University（北京语言大学）；LMU Munich（慕尼黑大学）；ELLIS Institute Finland（芬兰ELLIS研究所）；University of Turku（图尔库大学）；IBM Research AI, UAE（阿联酋IBM人工智能研究）；MBZUAI；Harbin Institute of Technology, Shenzhen（深圳哈尔滨工业大学）；Southern University of Science and Technology（南方科技大学）

Peiqin Lin, Chenyang Lyu, Wenjiang Luo, Haotian Ye, Md Mehrab Hossain, Chunlan Ma, Shaoxiong Ji, Younes Samih, Bo Zeng, Fan Jiang, Yuanbin Cao, Dilda Duisenbek,… 展开作者

Peiqin Lin, Chenyang Lyu, Wenjiang Luo, Haotian Ye, Md Mehrab Hossain, Chunlan Ma, Shaoxiong Ji, Younes Samih, Bo Zeng, Fan Jiang, Yuanbin Cao, Dilda Duisenbek, Adrian Neo Sau Xun, Daria Pozdniakova, Liubou Misevich, Nevena Marinković, Ngoc Gia Linh Nguyen, Thi Khanh Linh Do, Sarakmatak Sophy, Baotian Hu, Guanhua Chen, Gongbo Tang, Alham Fikri Aji, Longyue Wang, Weihua Luo

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出 CulturALL 基准测试，评估 LLM 在 grounded 任务中的多语言和多文化能力，包含 14 种语言 51 个地区 16 个主题的 2610 个样本，实验显示最佳 LLM 准确率仅 44.48%。

摘要：

大型语言模型（LLMs）已被全球部署，催生了大量衡量其多语言和多文化能力的基准测试。然而，这些基准测试侧重于通用语言理解和表面文化琐事，忽略了 grounded 任务——即模型必须在现实世界、上下文丰富的场景中推理——的评估。为填补这一空白，我们提出了 CulturALL，一个全面且具有挑战性的基准测试，用于评估 LLM 在 grounded 任务中的多语言和多文化能力。CulturALL 通过人机协作框架构建：专家标注者确保适当的难度和事实准确性，而 LLM 减轻人工工作量。通过整合多样化的来源，CulturALL 确保了全面的场景覆盖。每个项目都精心设计，以提供高水平的难度，使 CulturALL 具有挑战性。CulturALL 包含 2610 个样本，涵盖 14 种语言、51 个地区，分布在 16 个主题中，以捕捉 grounded 任务的完整广度。实验显示，最佳 LLM 在 CulturALL 上的准确率为 44.48%，凸显了显著的改进空间。

英文摘要：

Large language models (LLMs) are now deployed worldwide, inspiring a surge of benchmarks that measure their multilingual and multicultural abilities. However, these benchmarks prioritize generic language understanding or superficial cultural trivia, leaving the evaluation of grounded tasks -- where models must reason within real-world, context-rich scenarios -- largely unaddressed. To fill this gap, we present CulturALL, a comprehensive and challenging benchmark to assess LLMs' multilingual and multicultural competence on grounded tasks. CulturALL is built via a human--AI collaborative framework: expert annotators ensure appropriate difficulty and factual accuracy, while LLMs lighten the manual workload. By incorporating diverse sources, CulturALL ensures comprehensive scenario coverage. Each item is carefully designed to present a high level of difficulty, making CulturALL challenging. CulturALL contains 2,610 samples in 14 languages from 51 regions, distributed across 16 topics to capture the full breadth of grounded tasks. Experiments show that the best LLM achieves 44.48% accuracy on CulturALL, underscoring substantial room for improvement.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。