对话系统评估的心理学指标
Psychological Metrics for Dialog System Evaluation
AI总结：
本文提出五个基于心理学的可解释对话评估指标，并在多个数据集上验证其能提供与传统指标互补的新信息，从而提升对话系统评估的准确性和可解释性。
摘要：
我们提出了一套通过心理学基础的“人类”视角来评估对话系统的指标，在该视角下，对话智能体像人类一样表达多样的状态（如情绪）和特质（如个性）。我们提出了五个来自成熟心理学、对人类沟通和关系至关重要的可解释指标：情绪熵、语言风格与情绪匹配、宜人性和共情。这些指标可以（1）跨对话应用，也可以（2）在对话内部的轮次上应用。我们将这些心理学指标与七个最先进的传统指标（如BARTScore和BLEURT）在七个标准对话系统数据集上进行了比较。我们还引入了一个新的数据集——Three Bot Dialog Evaluation Corpus，该数据集由来自ChatGPT、GPT-3和BlenderBot的带标注对话组成。我们证明，所提出的指标提供了新颖的信息；它们与传统指标不相关，可用于有意义地比较对话系统，并在预测众包对话判断方面（在现有传统指标之外）带来更高的准确率。我们的心理学指标的可解释性和独特信号使其成为评估和改进对话系统的宝贵工具。
英文摘要：
We present metrics for evaluating dialog systems through a psychologically-grounded "human" lens in which conversational agents express a diversity of both states (e.g., emotion) and traits (e.g., personality), just as people do. We present five interpretable metrics from established psychology that are fundamental to human communication and relationships: emotional entropy, linguistic style and emotion matching, agreeableness, and empathy. These metrics can be applied (1) across dialogs and (2) on turns within dialogs. The psychological metrics are compared against seven state-of-the-art traditional metrics (e.g., BARTScore and BLEURT) on seven standard dialog system data sets. We also introduce a novel data set, the Three Bot Dialog Evaluation Corpus, which consists of annotated conversations from ChatGPT, GPT-3, and BlenderBot. We demonstrate that our proposed metrics offer novel information; they are uncorrelated with traditional metrics, can be used to meaningfully compare dialog systems, and lead to increased accuracy (beyond existing traditional metrics) in predicting crowd-sourced dialog judgements. The interpretability and unique signal of our psychological metrics make them a valuable tool for evaluating and improving dialog systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
