arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.17675cs.CL
收藏

用心理测量学评估大型语言模型

Evaluating Large Language Models with Psychometrics

University of Cambridge(剑桥大学);University of Notre Dame(圣母大学);Carnegie Mellon University(卡内基梅隆大学);Stanford University(斯坦福大学);Lehigh University(莱斯利大学)

Yuan Li, Yue Huang, Hongyi Wang, Ying Cheng, Xiangliang Zhang, James Zou, Lichao Sun

AI总结:

本文受心理测量学启发,构建了评估LLMs心理构念的综合基准,涵盖五个关键构念和13个数据集,发现自我报告与真实行为存在显著差异,为可靠评估提供见解。

摘要:

大型语言模型(LLMs)在解决各种任务中展现出卓越的能力,逐步演变为通用型助手。LLMs日益融入社会,引发了人们对它们是否表现出心理模式,以及这些模式在不同情境下是否保持一致的兴趣——这些问题可能加深对其行为的理解。受心理测量学启发,本文提出了一个用于量化LLMs心理构念的综合基准,涵盖心理维度识别、评估数据集设计以及带有结果验证的评估。我们的工作识别出五个关键心理构念——人格、价值观、情绪智力、心理理论和自我效能——并通过包含多样化场景和题型的一套13个数据集进行评估。我们发现了LLMs自我报告的特质与其在真实世界场景中的反应模式之间存在显著差异,揭示了其行为的复杂性。我们的研究结果还表明,一些原本为人类设计的基于偏好的测试无法从LLMs那里获得可靠的回答。本文对LLMs进行了全面的心理测量评估,为可靠评估以及人工智能和社会科学中的潜在应用提供了见解。

英文摘要:

Large Language Models (LLMs) have demonstrated exceptional capabilities in solving various tasks, progressively evolving into general-purpose assistants. The increasing integration of LLMs into society has sparked interest in whether they exhibit psychological patterns, and whether these patterns remain consistent across different contexts -- questions that could deepen the understanding of their behaviors. Inspired by psychometrics, this paper presents a {comprehensive benchmark for quantifying psychological constructs of LLMs}, encompassing psychological dimension identification, assessment dataset design, and assessment with results validation. Our work identifies five key psychological constructs -- personality, values, emotional intelligence, theory of mind, and self-efficacy -- assessed through a suite of 13 datasets featuring diverse scenarios and item types. We uncover significant discrepancies between LLMs' self-reported traits and their response patterns in real-world scenarios, revealing complexities in their behaviors. Our findings also show that some preference-based tests, originally designed for humans, could not solicit reliable responses from LLMs. This paper offers a thorough psychometric assessment of LLMs, providing insights into reliable evaluation and potential applications in AI and social sciences.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑