arXiv 2409.114042024-12-17cs.CLcs.AI

AraDiCE：大型语言模型中方言与文化能力的基准

AraDiCE: Benchmarks for Dialectal and Cultural Capabilities in LLMs

Qatar Computing Research Institute（卡塔尔计算研究所）；Qatar University（卡塔尔大学）；University of New Brunswick（新不伦瑞克大学）；American University of Sharjah（沙迦美国大学）

Basel Mousi, Nadir Durrani, Fatema Ahmad, Md. Arid Hasan, Maram Hasanain, Tameem Kabbani, Fahim Dalvi, Shammur Absar Chowdhury, Firoj Alam

备注：: Benchmarking, Culturally Informed, Large Language Models, Arabic NLP, LLMs, Arabic Dialect, Dialectal Benchmarking

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对阿拉伯语方言和文化代表性不足，提出AraDiCE基准，含7个方言数据集和约45K后期编辑样本，评估发现阿拉伯语专用模型优于多语言模型，但仍面临方言识别与生成挑战。

摘要：

阿拉伯语以其丰富的方言多样性，在大型语言模型中仍然代表性严重不足，尤其是在方言变体方面。我们通过引入七个与现代标准阿拉伯语（MSA）并行的方言合成数据集来解决这一差距，这些数据集是通过机器翻译（MT）结合人工后期编辑创建的。我们提出了AraDiCE，一个用于阿拉伯方言和文化评估的基准。我们评估了大型语言模型在方言理解和生成方面的能力，特别关注低资源阿拉伯方言。此外，我们首次引入了细粒度基准，旨在评估海湾、埃及和黎凡特地区的文化意识，为大型语言模型评估提供了新的维度。我们的研究结果表明，虽然像Jais和AceGPT这样的阿拉伯语专用模型在方言任务上优于多语言模型，但在方言识别、生成和翻译方面仍然存在重大挑战。这项工作贡献了约45K个后期编辑样本、一个文化基准，并强调了定制训练对于提高大型语言模型在捕捉多样化阿拉伯方言和文化背景细微差别方面性能的重要性。我们已发布了本研究中开发的方言翻译模型和基准（https://huggingface.co/datasets/QCRI/AraDiCE）。

英文摘要：

Arabic, with its rich diversity of dialects, remains significantly underrepresented in Large Language Models, particularly in dialectal variations. We address this gap by introducing seven synthetic datasets in dialects alongside Modern Standard Arabic (MSA), created using Machine Translation (MT) combined with human post-editing. We present AraDiCE, a benchmark for Arabic Dialect and Cultural Evaluation. We evaluate LLMs on dialect comprehension and generation, focusing specifically on low-resource Arabic dialects. Additionally, we introduce the first-ever fine-grained benchmark designed to evaluate cultural awareness across the Gulf, Egypt, and Levant regions, providing a novel dimension to LLM evaluation. Our findings demonstrate that while Arabic-specific models like Jais and AceGPT outperform multilingual models on dialectal tasks, significant challenges persist in dialect identification, generation, and translation. This work contributes $\approx$45K post-edited samples, a cultural benchmark, and highlights the importance of tailored training to improve LLM performance in capturing the nuances of diverse Arabic dialects and cultural contexts. We have released the dialectal translation models and benchmarks developed in this study (https://huggingface.co/datasets/QCRI/AraDiCE).

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。