大型语言模型在判断共情沟通方面何时具备可靠性
When Large Language Models are Reliable for Judging Empathic Communication
AI总结：
该研究通过对比专家、众包工作者与LLM在四种评估框架下对200段真实共情对话的标注信度，发现LLM共情判断可靠性接近专家水平且优于众包，验证了其在情感敏感场景的应用潜力。
摘要：
大型语言模型（LLM）擅长在文本对话中生成共情回应，但它们判断共情沟通细微差别的可靠性如何？我们通过比较专家、众包工作者和LLM对共情沟通的标注结果来探究这一问题，研究采用来自心理学、自然语言处理和传播学领域的四种评估框架，应用于200段真实对话——对话中一方倾诉个人问题，另一方提供支持。基于3150条专家标注、2844条众包标注和3150条LLM标注，我们评估了三类标注者之间的评分者间信度。研究发现，专家间一致性较高，但会因各框架子组件的清晰度、复杂度和主观性不同而存在差异。我们表明，与标准分类指标相比，专家一致性为衡量LLM表现提供了更具参考价值的基准。在全部四种框架下，LLM的表现始终接近这一专家级基准，且超过众包工作者的可靠性。这些结果表明，经特定任务与合适基准验证的LLM，可在情感敏感类应用（包括用作对话伙伴）中提升透明度并支持监督。
英文摘要：
Large language models (LLMs) excel at generating empathic responses in text-based conversations. But, how reliably do they judge the nuances of empathic communication? We investigate this question by comparing how experts, crowdworkers, and LLMs annotate empathic communication across four evaluative frameworks drawn from psychology, natural language processing, and communications applied to 200 real-world conversations where one speaker shares a personal problem and the other offers support. Drawing on 3,150 expert annotations, 2,844 crowd annotations, and 3,150 LLM annotations, we assess inter-rater reliability between these three annotator groups. We find that expert agreement is high but varies across the frameworks' sub-components depending on their clarity, complexity, and subjectivity. We show that expert agreement offers a more informative benchmark for contextualizing LLM performance than standard classification metrics. Across all four frameworks, LLMs consistently approach this expert level benchmark and exceed the reliability of crowdworkers. These results demonstrate how LLMs, when validated on specific tasks with appropriate benchmarks, can support transparency and oversight in emotionally sensitive applications including their use as conversational companions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
