arXiv 2504.125532025-04-17cs.CL

ELAB：波斯语大型语言模型对齐扩展基准

ELAB: Extensive LLM Alignment Benchmark in Persian Language

MCILAB；Sharif University of Technology（谢里夫理工大学）

Zahra Pourbahman, Fatemeh Rajabi, Mohammadhossein Sadeghi, Omid Ghahroodi, Somaye Bakhshaei, Arash Amini, Reza Kazemi, Mahdieh Soleymani Baghshah

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文针对波斯语大型语言模型（LLMs）的伦理对齐评估空白，构建了涵盖翻译、合成生成和自然收集数据的综合基准框架，系统评估其安全、公平和社会规范对齐表现，并提供公开排行榜。

摘要：

本文提出了一个综合评估框架，用于将波斯语大型语言模型（LLMs）与安全、公平和社会规范等关键伦理维度对齐。该框架通过适配波斯语的语言和文化语境，解决了现有LLM评估框架的空白。此基准创建了三类波斯语基准：（i）翻译数据、（ii）合成生成的新数据、（iii）自然收集的新数据。我们将Anthropic红队测试数据、AdvBench、HarmBench和DecodingTrust翻译成波斯语；还创建了ProhibiBench-fa、SafeBench-fa、FairBench-fa和SocialBench-fa作为新数据集，以应对本土文化中的有害和违禁内容；此外，收集了大规模数据集GuardBench-fa以考虑波斯文化规范。通过整合这些数据集，我们的工作建立了统一的波斯语LLM评估框架，提供了文化扎根的对齐评估新方法。我们对波斯语LLMs在安全（避免有害内容）、公平（减轻偏见）和社会规范（遵守文化接受行为）三个对齐方面进行了系统评估，并提供了公开可用的排行榜，在https://huggingface.co/spaces/MCILAB/LLM_Alignment_Evaluation对波斯语LLMs的安全、公平和社会规范进行基准测试。

英文摘要：

This paper presents a comprehensive evaluation framework for aligning Persian Large Language Models (LLMs) with critical ethical dimensions, including safety, fairness, and social norms. It addresses the gaps in existing LLM evaluation frameworks by adapting them to Persian linguistic and cultural contexts. This benchmark creates three types of Persian-language benchmarks: (i) translated data, (ii) new data generated synthetically, and (iii) new naturally collected data. We translate Anthropic Red Teaming data, AdvBench, HarmBench, and DecodingTrust into Persian. Furthermore, we create ProhibiBench-fa, SafeBench-fa, FairBench-fa, and SocialBench-fa as new datasets to address harmful and prohibited content in indigenous culture. Moreover, we collect extensive dataset as GuardBench-fa to consider Persian cultural norms. By combining these datasets, our work establishes a unified framework for evaluating Persian LLMs, offering a new approach to culturally grounded alignment evaluation. A systematic evaluation of Persian LLMs is performed across the three alignment aspects: safety (avoiding harmful content), fairness (mitigating biases), and social norms (adhering to culturally accepted behaviors). We present a publicly available leaderboard that benchmarks Persian LLMs with respect to safety, fairness, and social norms at: https://huggingface.co/spaces/MCILAB/LLM_Alignment_Evaluation.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。