情绪麻木还是共情？使用EmotionBench评估大语言模型的感受方式
Emotionally Numb or Empathetic? Evaluating How LLMs Feel Using EmotionBench
AI总结：
该研究借鉴心理学情绪评价理论，构建含400余情境的EmotionBench测试框架，评估7款不同规模的最新大语言模型的共情能力，发现其有一定情绪回应能力但与人类情绪行为对齐不足，相关资源已公开。
摘要：
评估大语言模型（LLM）的拟人化能力在当下的讨论中变得愈发重要。我们借鉴心理学中的情绪评价理论，提出评估LLM的共情能力，即它们在面对特定情境时的感受会如何变化。经过仔细全面的调研，我们收集了一个包含400余个情境的数据集，这些情境已被证实能有效唤起本研究核心的八种情绪。我们将这些情境划分为36个因素，开展了一项涉及全球1200余名受试者的人类评估。以人类评估结果为参照，我们的评估涵盖7个LLM，包括商业模型与开源模型，覆盖不同模型规模，包含最新迭代版本，如GPT-4、Mixtral-8x22B和LLaMA-3.1。我们发现，尽管存在若干不一致之处，LLM通常能对特定情境做出恰当回应。但它们在与人类情绪行为的对齐上仍有不足，且无法在相似情境间建立关联。我们收集的情境数据集、人类评估结果以及测试框架EmotionBench的代码已在https://github.com/CUHK-ARISE/EmotionBench公开。
英文摘要：
Evaluating Large Language Models' (LLMs) anthropomorphic capabilities has become increasingly important in contemporary discourse. Utilizing the emotion appraisal theory from psychology, we propose to evaluate the empathy ability of LLMs, i.e., how their feelings change when presented with specific situations. After a careful and comprehensive survey, we collect a dataset containing over 400 situations that have proven effective in eliciting the eight emotions central to our study. Categorizing the situations into 36 factors, we conduct a human evaluation involving more than 1,200 subjects worldwide. With the human evaluation results as references, our evaluation includes seven LLMs, covering both commercial and open-source models, including variations in model sizes, featuring the latest iterations, such as GPT-4, Mixtral-8x22B, and LLaMA-3.1. We find that, despite several misalignments, LLMs can generally respond appropriately to certain situations. Nevertheless, they fall short in alignment with the emotional behaviors of human beings and cannot establish connections between similar situations. Our collected dataset of situations, the human evaluation results, and the code of our testing framework, i.e., EmotionBench, are publicly available at https://github.com/CUHK-ARISE/EmotionBench.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
