价值罗盘基准：一个用于基础且有效评估大语言模型价值观的平台
Value Compass Benchmarks: A Platform for Fundamental and Validated Evaluation of LLMs Values
AI总结：
本文提出价值罗盘基准，通过基本价值观澄清、生成式演化评估和多元加权对齐度量，系统解决LLMs价值观评估中的价值澄清、有效性和多元性三大挑战。
摘要：
随着大语言模型（LLMs）取得显著突破，使其价值观与人类对齐已成为其负责任发展和定制化应用的当务之急。然而，目前仍缺乏能够满足三个理想目标的LLMs价值观评估。（1）价值澄清：我们期望精确且全面地澄清LLMs的潜在价值观，而当前评估仅狭隘地关注偏见和毒性等安全风险。（2）评估有效性：现有的静态开源基准容易受到数据污染，并随着LLMs的演进迅速过时。此外，这些判别式评估揭示的是LLMs关于价值观的知识，而非对LLMs行为符合价值观程度的有效评估。（3）价值多元性：人类价值观在个体和文化之间具有多元性，但在衡量LLMs价值对齐时，这一点在很大程度上被忽视。为应对这些挑战，我们提出了价值罗盘基准（Value Compass Benchmarks），并设计了三个相应的模块。它（i）将评估建立在动机上不同的基本价值观之上，以从整体视角澄清LLMs的潜在价值观；（ii）应用生成式演化评估框架，采用自适应测试项目来应对不断演进的LLMs，并在现实场景中从行为直接识别价值观；（iii）提出一种度量指标，将LLMs与特定价值观的对齐程度量化为多个维度上的加权和，其中权重由多元价值观决定。
英文摘要：
As Large Language Models (LLMs) achieve remarkable breakthroughs, aligning their values with humans has become imperative for their responsible development and customized applications. However, there still lack evaluations of LLMs values that fulfill three desirable goals. (1) Value Clarification: We expect to clarify the underlying values of LLMs precisely and comprehensively, while current evaluations focus narrowly on safety risks such as bias and toxicity. (2) Evaluation Validity: Existing static, open-source benchmarks are prone to data contamination and quickly become obsolete as LLMs evolve. Additionally, these discriminative evaluations uncover LLMs' knowledge about values, rather than valid assessments of LLMs' behavioral conformity to values. (3) Value Pluralism: The pluralistic nature of human values across individuals and cultures is largely ignored in measuring LLMs value alignment. To address these challenges, we presents the Value Compass Benchmarks, with three correspondingly designed modules. It (i) grounds the evaluation on motivationally distinct \textit{basic values to clarify LLMs' underlying values from a holistic view; (ii) applies a \textit{generative evolving evaluation framework with adaptive test items for evolving LLMs and direct value recognition from behaviors in realistic scenarios; (iii) propose a metric that quantifies LLMs alignment with a specific value as a weighted sum over multiple dimensions, with weights determined by pluralistic values.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
