ArabLegalEval：用于评估大型语言模型阿拉伯语法律知识的多任务基准
ArabLegalEval: A Multitask Benchmark for Assessing Arabic Legal Knowledge in Large Language Models
AI总结：
针对阿拉伯语法律知识评估的空白，提出多任务基准ArabLegalEval，涵盖沙特法律文档与合成问题，评测GPT-4和Jais等模型，并探索上下文学习与自动验证生成方法，以推动该领域研究。
摘要：
大型语言模型（LLMs）的快速发展已显著提升了各种自然语言处理任务的性能。然而，对LLMs法律知识的评估，尤其是在阿拉伯语等非英语语言中，仍探索不足。为弥补这一空白，我们推出了ArabLegalEval，一个用于评估LLMs阿拉伯语法律知识的多任务基准数据集。受MMLU和LegalBench数据集的启发，ArabLegalEval包含源自沙特法律文件和合成问题的多个任务。在本研究中，我们旨在分析解决阿拉伯语法律问题所需的能力，并对最先进的LLMs进行基准测试。我们探讨了上下文学习的影响，并研究了各种评估方法。此外，我们探索了通过自动验证生成问题的工作流程，以提高数据集的质量。我们对多语言和以阿拉伯语为中心的LLMs（如GPT-4和Jais）进行了基准测试。我们还分享了创建数据集和验证的方法，这些方法可推广到其他领域。我们希望通过发布ArabLegalEval数据集和代码（https://github.com/Thiqah/ArabLegalEval）来加速阿拉伯语法律领域的人工智能研究。
英文摘要：
The rapid advancements in Large Language Models (LLMs) have led to significant improvements in various natural language processing tasks. However, the evaluation of LLMs' legal knowledge, particularly in non-English languages such as Arabic, remains under-explored. To address this gap, we introduce ArabLegalEval, a multitask benchmark dataset for assessing the Arabic legal knowledge of LLMs. Inspired by the MMLU and LegalBench datasets, ArabLegalEval consists of multiple tasks sourced from Saudi legal documents and synthesized questions. In this work, we aim to analyze the capabilities required to solve legal problems in Arabic and benchmark the performance of state-of-the-art LLMs. We explore the impact of in-context learning and investigate various evaluation methods. Additionally, we explore workflows for generating questions with automatic validation to enhance the dataset's quality. We benchmark multilingual and Arabic-centric LLMs, such as GPT-4 and Jais, respectively. We also share our methodology for creating the dataset and validation, which can be generalized to other domains. We hope to accelerate AI research in the Arabic Legal domain by releasing the ArabLegalEval dataset and code: https://github.com/Thiqah/ArabLegalEval
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
