AraTrust：阿拉伯语大语言模型可信度评估
AraTrust: An Evaluation of Trustworthiness for LLMs in Arabic
AI总结：
针对阿拉伯语LLMs缺乏可信度评估基准的问题，提出首个全面基准AraTrust，含522道人工题，评估发现GPT-4最可信，开源模型如AceGPT 7B和Jais 13B得分低于60%。
摘要：
人工智能（AI）系统的快速进步和广泛接受凸显了理解AI相关能力与潜在风险的迫切需求。鉴于阿拉伯语在语言上的复杂性、文化上的丰富性以及在AI研究中的代表性不足，迫切需要关注大语言模型（LLMs）在阿拉伯语相关任务上的性能与安全性。尽管其开发取得了一些进展，但缺乏全面的可信度评估基准，这对准确评估和提高LLMs在阿拉伯语提示下的安全性构成了重大挑战。在本文中，我们介绍了AraTrust，这是首个针对阿拉伯语LLMs的全面可信度基准。AraTrust包含522道人工编写的多项选择题，涉及真实性、伦理、安全、身体健康、心理健康、不公平、非法活动、隐私和冒犯性语言等多个维度。我们针对该基准评估了一系列LLMs的可信度。GPT-4是最可信的LLM，而开源模型，特别是AceGPT 7B和Jais 13B，在我们的基准中难以达到60%的得分。
英文摘要：
The swift progress and widespread acceptance of artificial intelligence (AI) systems highlight a pressing requirement to comprehend both the capabilities and potential risks associated with AI. Given the linguistic complexity, cultural richness, and underrepresented status of Arabic in AI research, there is a pressing need to focus on Large Language Models (LLMs) performance and safety for Arabic-related tasks. Despite some progress in their development, there is a lack of comprehensive trustworthiness evaluation benchmarks, which presents a major challenge in accurately assessing and improving the safety of LLMs when prompted in Arabic. In this paper, we introduce AraTrust, the first comprehensive trustworthiness benchmark for LLMs in Arabic. AraTrust comprises 522 human-written multiple-choice questions addressing diverse dimensions related to truthfulness, ethics, safety, physical health, mental health, unfairness, illegal activities, privacy, and offensive language. We evaluated a set of LLMs against our benchmark to assess their trustworthiness. GPT-4 was the most trustworthy LLM, while open-source models, particularly AceGPT 7B and Jais 13B, struggled to achieve a score of 60% in our benchmark.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
