arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.14903cs.AI
收藏

GIEBench:面向大语言模型基于群体身份共情能力的整体性评估

GIEBench: Towards Holistic Evaluation of Group Identity-based Empathy for Large Language Models

Beijing University of Posts and Telecommunications(北京邮电大学);Tianjin University(天津大学);M-A-P;Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Waterloo(滑铁卢大学)

Leyan Wang, Yonggang Jin, Tianhao Shen, Tianyu Zheng, Xinrun Du, Chenchen Zhang, Wenhao Huang, Jiaheng Liu, Shi Wang, Ge Zhang, Liuyu Xiang, Zhaofeng He

AI总结:

针对现有大语言模型共情评估忽略群体身份背景的问题,研究推出覆盖11个维度、97个群体身份的GIEBench基准,测试发现23款LLM在无明确视角指示时无法对不同身份展现同等共情,凸显多元价值对齐的必要性。

摘要:

随着大语言模型(LLMs)持续发展并得到广泛应用,LLMs对不同群体身份展现共情、理解其视角的能力愈发被视为关键特性。现有多数LLM共情评估基准主要聚焦悲伤、痛苦等人类普遍情绪,往往忽略个体的群体身份背景。为填补这一空白,我们推出GIEBench(基于群体身份的共情评估基准),这一综合性基准包含11个身份维度,覆盖97个群体身份,共计999道与特定群体身份相关的单选题。GIEBench旨在评估LLM在面对性别、年龄、职业、种族等特定群体身份时的共情能力,重点考察其从对应群体立场出发作出回应的能力,为面向不同身份用户定制的共情型LLM应用的持续开发提供支撑。我们对23款LLM的评估显示,尽管这些LLM能够理解不同的身份立场,但在没有明确指示采用对应视角的情况下,它们无法始终对所有身份展现同等程度的共情。这一结果凸显了需要提升LLM与多元价值观的对齐程度,以更好适配人类身份的多面性。我们的数据集可在https://github.com/GIEBench/GIEBench获取。

英文摘要:

As large language models (LLMs) continue to develop and gain widespread application, the ability of LLMs to exhibit empathy towards diverse group identities and understand their perspectives is increasingly recognized as critical. Most existing benchmarks for empathy evaluation of LLMs focus primarily on universal human emotions, such as sadness and pain, often overlooking the context of individuals' group identities. To address this gap, we introduce GIEBench, a comprehensive benchmark that includes 11 identity dimensions, covering 97 group identities with a total of 999 single-choice questions related to specific group identities. GIEBench is designed to evaluate the empathy of LLMs when presented with specific group identities such as gender, age, occupation, and race, emphasizing their ability to respond from the standpoint of the identified group. This supports the ongoing development of empathetic LLM applications tailored to users with different identities. Our evaluation of 23 LLMs revealed that while these LLMs understand different identity standpoints, they fail to consistently exhibit equal empathy across these identities without explicit instructions to adopt those perspectives. This highlights the need for improved alignment of LLMs with diverse values to better accommodate the multifaceted nature of human identities. Our datasets are available at https://github.com/GIEBench/GIEBench.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑