AraHalluEval：面向阿拉伯语大语言模型的细粒度幻觉评估框架
AraHalluEval: A Fine-grained Hallucination Evaluation Framework for Arabic LLMs
AI总结：
本文提出AraHalluEval，首个针对阿拉伯语LLM的细粒度幻觉评估框架，在生成式问答和摘要任务上评估12个模型，发现事实性幻觉更普遍，且阿拉伯语模型Allam表现优于多语言模型。
摘要：
近年来，关于大语言模型（LLMs）幻觉的大量研究主要集中在英语上。尽管多语言和阿拉伯语专用LLM的数量不断增加，但在阿拉伯语背景下评估LLM的幻觉仍相对未被充分探索。鉴于阿拉伯语在许多地区的广泛使用及其在全球交流和媒体中的重要性，这一知识空白尤为紧迫。本文首次对阿拉伯语和多语言LLM在两个关键的阿拉伯语自然语言生成任务上进行了全面的幻觉评估：生成式问答（GQA）和摘要生成。本研究共评估了12个LLM，包括4个阿拉伯语预训练模型、4个多语言模型和4个基于推理的模型。为了评估LLM输出的事实一致性和忠实度，我们开发了一个细粒度的幻觉评估框架，该框架由12个细粒度幻觉指标组成，这些指标代表了每个任务的不同特征。结果表明，在所有模型和任务中，事实性幻觉比忠实性错误更为普遍。值得注意的是，阿拉伯语预训练模型Allam始终表现出比多语言模型更低的幻觉率，并与基于推理的模型表现相当。代码可在以下网址获取：https://github.com/aishaalansari57/AraHalluEval
英文摘要：
Recently, extensive research on the hallucination of the large language models (LLMs) has mainly focused on the English language. Despite the growing number of multilingual and Arabic-specific LLMs, evaluating LLMs' hallucination in the Arabic context remains relatively underexplored. The knowledge gap is particularly pressing given Arabic's widespread use across many regions and its importance in global communication and media. This paper presents the first comprehensive hallucination evaluation of Arabic and multilingual LLMs on two critical Arabic natural language generation tasks: generative question answering (GQA) and summarization. This study evaluates a total of 12 LLMs, including 4 Arabic pre-trained models, 4 multilingual models, and 4 reasoning-based models. To assess the factual consistency and faithfulness of LLMs' outputs, we developed a fine-grained hallucination evaluation framework consisting of 12 fine-grained hallucination indicators that represent the varying characteristics of each task. The results reveal that factual hallucinations are more prevalent than faithfulness errors across all models and tasks. Notably, the Arabic pre-trained model Allam consistently demonstrates lower hallucination rates than multilingual models and a comparative performance with reasoning-based models. The code is available at: https://github.com/aishaalansari57/AraHalluEval
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
