超越多项选择题：一个包含方言变体的开放性阿拉伯文化问答基准测试
Beyond MCQ: An Open-Ended Arabic Cultural QA Benchmark with Dialect Variants
AI总结：
本文提出了一种方法，将现代标准阿拉伯语的多项选择题翻译成英文和阿拉伯方言，并转换为开放性问题，评估不同模型在不同设置下的表现，发现模型在阿拉伯方言上表现欠佳，CoT方法提高了正确性但指标混杂。
摘要：
大型语言模型（LLMs）越来越多地用于回答日常问题，但其在文化相关和方言内容上的表现因语言而异。我们提出了一种全面的方法，包括将现代标准阿拉伯语的多项选择题（MCQs）翻译成英文和几种阿拉伯方言，将其转换为开放性问题（OEQs），在MCQ和OEQ设置下评估多种零样本和微调LLMs，生成链式思考（CoT）推理以微调模型进行逐步推理。使用此方法，我们扩展了一个现有数据集，使问答在多种语言变体中平行对齐，使其成为首个此类数据集。我们对开放和闭合模型进行了广泛实验。我们的发现显示：（i）模型在阿拉伯方言上表现不佳，揭示了文化相关和方言特定知识的持续差距；（ii）以阿拉伯为中心的模型在MCQs上表现良好，但在OEQs上挣扎；（iii）CoT提高了判断正确性，但导致混合的n-gram基于指标。开发的数据集将被公开发布，以支持对文化上和语言上包容性评估的进一步研究。
英文摘要：
Large Language Models (LLMs) are increasingly used to answer everyday questions, yet their performance on culturally grounded and dialectal content remains uneven across languages. We propose a comprehensive method that (i) translates Modern Standard Arabic (MSA) multiple-choice questions (MCQs) into English and several Arabic dialects, (ii) converts them into open-ended questions (OEQs), (iii) benchmarks a range of zero-shot and fine-tuned LLMs under both MCQ and OEQ settings, and (iv) generates chain-of-thought (CoT) rationales to fine-tune models for step-by-step reasoning. Using this method, we extend an existing dataset in which QAs are parallelly aligned across multiple language varieties, making it, to our knowledge, the first of its kind. We conduct extensive experiments with both open and closed models. Our findings show that (i) models underperform on Arabic dialects, revealing persistent gaps in culturally grounded and dialect-specific knowledge; (ii) Arabic-centric models perform well on MCQs but struggle with OEQs; and (iii) CoT improves judged correctness while yielding mixed n-gram-based metrics. The developed dataset will be publicly released to support further research on culturally and linguistically inclusive evaluation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
