阿拉伯语言模型的文化基准测试：标准阿拉伯语与方言对话
Cultural Benchmarking of LLMs in Standard and Dialectal Arabic Dialogues
AI总结：
本文提出ArabCulture-Dialogue数据集，涵盖13国阿拉伯语及方言，通过三种任务评估模型在文化推理、翻译和方言引导生成中的表现，发现方言设置下模型性能仍低于标准阿拉伯语。
摘要：
本文提出了ArabCulture-Dialogue数据集，涵盖13国阿拉伯语及方言，通过三种任务评估模型在文化推理、翻译和方言引导生成中的表现，发现方言设置下模型性能仍低于标准阿拉伯语。
英文摘要：
There is a significant gap in evaluating cultural reasoning in LLMs using conversational datasets that capture culturally rich and dialectal contexts. Most Arabic benchmarks focus on short text snippets in Modern Standard Arabic (MSA), overlooking the cultural nuances that naturally arise in dialogues. To address this gap, we introduce ArabCulture-Dialogue, a culturally grounded conversational dataset covering 13 Arabic-speaking countries, in both MSA and each country's respective dialect, spanning 12 daily-life topics and 54 fine-grained subtopics. We utilize the dataset to form three benchmarking tasks: (i) multiple-choice cultural reasoning, (ii) machine translation between MSA and dialects, and (iii) dialect-steering generation. Our experiments indicate that the performance gap between MSA and Arabic dialects still exists, whereby the models perform worse on all three tasks in the dialectal setup, compared to the MSA one.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
