arXiv 2605.217392026-05-27cs.AI

AttuneBench: 基于对话的LLM情商基准测试

AttuneBench: A Conversation-Based Benchmark for LLM Emotional Intelligence

Pareto；Thoughtful；University of Pennsylvania（宾夕法尼亚大学）

Kate M. Lubrano, Faisal Sayed, Ankita Rathod, Akshansh, Craver Corbyn Thomas-Smith, Mark E. Whiting, Karina Nguyen

备注：: v2: Updated def_18 and def_20 supplemental figures to cover all 11 evaluated models (previously 9). Removed redundant supplemental figures. Corrected select captions (color descriptions, chance baselines, figure-content mismatches). No changes to experimental results, numerical claims, or conclusions

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出AttuneBench基准，基于200个真实多轮人机对话，评估LLM在情绪识别、行为分类、偏好预测和响应质量等方面的情商能力，发现这些能力相互独立且偏好对齐和响应质量更具区分性。

摘要：

情商（EI），即感知、理解并恰当回应他人情绪状态的能力，是人类交流的核心，随着LLM在日常生活中承担对话角色，评估其情商日益重要。现有的EI基准依赖于合成提示、单轮案例或第三方标注。这些方法不能直接衡量模型在真实对话过程中如何推断和回应参与者的情绪状态。我们引入AttuneBench，一个基于200个真实多轮人机对话的基准，其中参与者与匿名LLM对话，并逐轮标注其情绪状态、模型行为以及他们偏好的回应。在11个评估模型中，我们发现模型在情绪识别、行为分类、偏好预测和评判响应质量上的排名基本独立，表明情商行为可分解为可分离的能力。偏好对齐和响应质量判断比情绪标签准确性更具模型区分性。这些结果表明，情商行为需要预测特定用户在上下文中想要什么样的回应，这一区别可能被总体评分掩盖，而单轮或合成格式无法跨轮直接捕捉。AttuneBench提供了一个评估这些能力以及诊断模型在情绪显著对话中的特定优势和失败模式的框架。

英文摘要：

Emotional intelligence (EI), the ability to perceive, understand, and respond appropriately to others' emotional states, is central to human communication, and increasingly important to assess as LLMs assume conversational roles in everyday life. Existing EI benchmarks rely on synthetic prompts, single-turn cases, or third-party annotation. These approaches do not directly measure how models infer and respond to a participant's emotional state over the course of a real conversation. We introduce AttuneBench, a benchmark grounded in 200 genuine multi-turn human-model conversations in which participants conversed with anonymized LLMs and provided turn-by-turn annotations of their emotional state, the model's behavior, and their preferred responses. Across 11 evaluated models, we find that model rankings on emotion recognition, behavioral classification, preference prediction, and judged response quality are largely independent, indicating that emotionally intelligent behavior decomposes into separable capabilities. Preference alignment and response-quality judgments are substantially more model-discriminating than emotion-label accuracy. These results indicate that emotionally intelligent behavior requires predicting what kind of response a specific user wants in context, a distinction that aggregate scoring can obscure and that single-turn or synthetic formats cannot directly capture across turns. AttuneBench provides a framework for assessing each of these capabilities and for diagnosing model-specific strengths and failure modes in emotionally salient conversation.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。