ESC-Judge：一个用于比较情感支持对话智能体的框架
ESC-Judge: A Framework for Comparing Emotional Support Conversational Agents
AI总结：
针对情感支持大语言模型缺乏可扩展、有理论依据的评估方法的问题，研究提出基于“探索-领悟-行动”咨询模型的端到端自动化评估框架ESC-Judge，其判断准确率接近博士级标注员水平，相关资源已公开。
摘要：
大语言模型（LLMs）正越来越多地为心理健康聊天机器人提供支持，但该领域仍缺乏一种可扩展、有理论依据的方法来判定哪款模型最适合部署。我们提出了ESC-Judge，这是首个端到端评估框架，它具备两大特性：（i）基于Clara Hill已确立的“探索-领悟-行动”咨询模型，对情感支持LLMs进行一对一对比，提供结构化且可解释的性能表现视图；（ii）实现全流程自动化的大规模评估。ESC-Judge分三个阶段运行：第一，通过采样压力源、性格、生活经历等经实证验证的显著属性，合成真实的求助者角色；第二，让两个候选支持智能体分别与同一角色进行对话会话，分离出模型特有的策略；第三，调用专门的评判LLM，针对涵盖探索、领悟、行动全维度的、以评分细则为锚点的各项技能给出两两偏好判断。在我们的研究中，ESC-Judge在85%的探索类决策、83%的领悟类决策和86%的行动类决策上与博士级标注员的判断一致，以极低的成本实现了人类水平的可靠性。我们已公开所有代码、提示词、合成角色、对话记录和评判脚本，以推动情感支持AI领域的透明化进展。
英文摘要：
Large language models (LLMs) increasingly power mental-health chatbots, yet the field still lacks a scalable, theory-grounded way to decide which model is most effective to deploy. We present ESC-Judge, the first end-to-end evaluation framework that (i) grounds head-to-head comparisons of emotional-support LLMs in Clara Hill's established Exploration-Insight-Action counseling model, providing a structured and interpretable view of performance, and (ii) fully automates the evaluation pipeline at scale. ESC-Judge operates in three stages: first, it synthesizes realistic help-seeker roles by sampling empirically salient attributes such as stressors, personality, and life history; second, it has two candidate support agents conduct separate sessions with the same role, isolating model-specific strategies; and third, it asks a specialized judge LLM to express pairwise preferences across rubric-anchored skills that span the Exploration, Insight, and Action spectrum. In our study, ESC-Judge matched PhD-level annotators on 85 percent of Exploration, 83 percent of Insight, and 86 percent of Action decisions, demonstrating human-level reliability at a fraction of the cost. All code, prompts, synthetic roles, transcripts, and judgment scripts are released to promote transparent progress in emotionally supportive AI.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
