EPT基准：大型语言模型中波斯语可信度的评估
EPT Benchmark: Evaluation of Persian Trustworthiness in Large Language Models
AI总结：
本研究提出EPT基准，用于评估大型语言模型在波斯语语境下的可信度，涵盖真实性、安全性等六个维度，发现安全性缺陷显著，并公开数据集以促进文化负责任AI的发展。
摘要：
大型语言模型（LLMs）在先进深度学习架构的支持下，基于大规模数据集进行训练，已在广泛的语言任务中展现出卓越的性能，成为现代人工智能技术的基石。然而，确保其可信度仍是一个关键挑战，因为可靠性不仅对于准确性能至关重要，而且对于维护伦理、文化和社会价值观也必不可少。对训练数据的仔细对齐以及基于文化的评估标准对于开发负责任的人工智能系统至关重要。在本研究中，我们引入了EPT（波斯语可信度评估）指标，这是一个具有文化意识的基准，专门设计用于评估LLMs在六个关键方面的可信度：真实性、安全性、公平性、鲁棒性、隐私性和伦理对齐。我们整理了一个带标签的数据集，并使用基于LLM的自动评估和人工评估，评估了多个领先模型的性能，包括ChatGPT、Claude、DeepSeek、Gemini、Grok、LLaMA、Mistral和Qwen。我们的结果揭示了安全性维度的显著缺陷，强调了迫切需要关注模型行为的这一关键方面。此外，我们的发现为这些模型与波斯伦理文化价值观的对齐提供了宝贵的见解，并指出了推进可信和负责任人工智能的关键差距与机遇。该数据集公开可用，网址为：https://github.com/Rezamirbagheri110/EPT-Benchmark。
英文摘要：
Large Language Models (LLMs), trained on extensive datasets using advanced deep learning architectures, have demonstrated remarkable performance across a wide range of language tasks, becoming a cornerstone of modern AI technologies. However, ensuring their trustworthiness remains a critical challenge, as reliability is essential not only for accurate performance but also for upholding ethical, cultural, and social values. Careful alignment of training data and culturally grounded evaluation criteria are vital for developing responsible AI systems. In this study, we introduce the EPT (Evaluation of Persian Trustworthiness) metric, a culturally informed benchmark specifically designed to assess the trustworthiness of LLMs across six key aspects: truthfulness, safety, fairness, robustness, privacy, and ethical alignment. We curated a labeled dataset and evaluated the performance of several leading models - including ChatGPT, Claude, DeepSeek, Gemini, Grok, LLaMA, Mistral, and Qwen - using both automated LLM-based and human assessments. Our results reveal significant deficiencies in the safety dimension, underscoring the urgent need for focused attention on this critical aspect of model behavior. Furthermore, our findings offer valuable insights into the alignment of these models with Persian ethical-cultural values and highlight critical gaps and opportunities for advancing trustworthy and culturally responsible AI. The dataset is publicly available at: https://github.com/Rezamirbagheri110/EPT-Benchmark.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
