潘查希拉困境：基于潘查希拉对印度尼西亚人类价值困境的大语言模型评估

中文摘要 AI 辅助

大语言模型（LLMs）的价值对齐对于确保其回答符合人类意图和价值偏好至关重要。然而，大多数价值对齐评估集中在西方或普遍价值上，基于特定国家价值体系的评估仍然稀缺。本文引入了潘查希拉困境，这是一个由1834个源自印尼新闻的问题组成的评估数据集，按潘查希拉的五个价值观分类：宗教、人道、团结、民主和社会正义。该数据集反映了印尼的日常生活，适用于衡量为印尼部署的大语言模型的价值对齐。为确保更严格的评估，选择了包含困境的场景。数据集由母语人士校对，并由5名不同的印尼公民回答。在该数据集上评估了50个封闭和开源大语言模型。结果显示，所有评估的大语言模型的概率匹配分数（PMS）均低于0.5，最大投票一致性分数（MVAS）低于0.72。按每个价值观比较，大语言模型在宗教和团结困境案例中大多表现不佳。这凸显了在捕捉印尼价值观方面的重大差距。该数据集可在指定网址公开获取。

英文摘要

The value alignment of large language models (LLMs) is crucial for ensuring responses align with human intention and value preferences. However, most evaluations of value alignment focus on Western or universal values, while assessments grounded in the value systems of specific countries remain scarce. In this paper, we introduce Pancasila-Dilemmas, an evaluation dataset of 1,834 questions derived from Indonesian news, classified by 5 values of Pancasila: Religion, Humanity, Unity, Democracy, and Social Justice. This dataset reflects daily life in Indonesia, making it suitable for measuring the value alignment of LLMs deployed for Indonesia. To ensure a more rigorous evaluation, we choose scenarios containing dilemmas. The dataset is proofread by native speakers and answered by 5 diverse Indonesian citizens. We evaluate 50 closed- and open-source LLMs on our dataset. Results reveal that all evaluated LLMs achieves less than 0.5 Probability Match Score (PMS) and 0.72 Max-Vote Agreement Score (MVAS). Compared by each values, LLMs mostly struggle in Religion and Unity dilemma cases. This highlights a significant gap in capturing Indonesian values. The dataset is publicly available at https://github.com/tjunlp-lab/Pancasila-Dilemmas.

发表机构

• Universitas Universal（印尼环球大学）
• Beijing Language and Culture University（北京语言大学）
• Tianjin University（天津大学）
• School of Artificial Intelligence, Tianjin University（天津大学人工智能学院）
• School of Computer Science and Technology, Tianjin University（天津大学计算机科学与技术学院）

机构由 AI 辅助整理，请以论文原文为准。