arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.04214cs.CL
收藏

ValueBench:迈向全面评估大语言模型价值取向与价值理解

ValueBench: Towards Comprehensively Evaluating Value Orientations and Understanding of Large Language Models

Peking University(北京大学);National Key Laboratory of General Artificial Intelligence(通用人工智能国家重点实验室);School of Intelligence Science and Technology, Peking University(北京大学智能科学与技术学院);Department of Sociology, Peking University(北京大学社会学系);School of Psychological and Cognitive Sciences, Peking University(北京大学心理与认知科学学院);PKU-Wuhan Institute for Artificial Intelligence(北京大学武汉人工智能研究院)

Yuanyi Ren, Haoran Ye, Hanjun Fang, Xin Zhang, Guojie Song

AI总结:

本文提出首个综合性心理测量基准ValueBench,通过44个量表、453个价值维度及真实交互流程,系统评估大语言模型的价值取向与价值理解,并在六个代表性模型上验证其接近专家结论的能力。

摘要:

大语言模型(LLMs)正在变革多个领域,并作为人类代理获得日益增长的影响力。这一发展凸显了评估LLMs价值取向与价值理解的迫切需求,以确保其负责任地融入面向公众的应用。本工作提出了ValueBench,这是首个用于评估LLMs价值取向和价值理解的综合性心理测量基准。ValueBench从44个既有心理测量量表中收集数据,涵盖453个多维价值维度。我们提出了一条基于真实人类-AI交互的评估流程,用于探测价值取向,并提出了在开放式价值空间中评估价值理解的新任务。通过在六个代表性LLMs上进行的大量实验,我们揭示了它们共有且独特的价值取向,并展示了它们在价值相关的抽取与生成任务中接近专家结论的能力。ValueBench可在https://github.com/Value4AI/ValueBench公开获取。

英文摘要:

Large Language Models (LLMs) are transforming diverse fields and gaining increasing influence as human proxies. This development underscores the urgent need for evaluating value orientations and understanding of LLMs to ensure their responsible integration into public-facing applications. This work introduces ValueBench, the first comprehensive psychometric benchmark for evaluating value orientations and value understanding in LLMs. ValueBench collects data from 44 established psychometric inventories, encompassing 453 multifaceted value dimensions. We propose an evaluation pipeline grounded in realistic human-AI interactions to probe value orientations, along with novel tasks for evaluating value understanding in an open-ended value space. With extensive experiments conducted on six representative LLMs, we unveil their shared and distinctive value orientations and exhibit their ability to approximate expert conclusions in value-related extraction and generation tasks. ValueBench is openly accessible at https://github.com/Value4AI/ValueBench.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑