PMIScore: 一种无监督量化对话参与度的方法
PMIScore: An Unsupervised Approach to Quantify Dialogue Engagement
AI总结：
本文提出PMIScore，一种无监督方法，通过点互信息量化对话参与度，利用大语言模型生成对话对并训练神经网络，验证了其在合成和真实数据集上的有效性。
摘要：
高对话参与度是有效对话的关键指标。可靠的参与度测量有助于评估大语言模型、提升人机交互效果或改善个人沟通技能。然而，量化参与度具有挑战性，因为其主观性且缺乏'金标准'。本文提出PMIScore，一种高效的无监督方法，利用点互信息（PMI），即在对话历史条件下生成响应的概率。因此，PMIScore提供了参与度的清晰解释。由于直接计算PMI因对话复杂性而不可行，PMIScore通过双重发散形式学习PMI。该算法包括生成正负对话对、通过大语言模型（LLMs）提取嵌入并向小神经网络施加互信息损失函数进行训练。我们在合成和真实数据集上验证了PMIScore。结果表明，PMIScore在PMI估计中的有效性以及PMI指标本身的合理性。
英文摘要：
High dialogue engagement is a crucial indicator of an effective conversation. A reliable measure of engagement could help benchmark large language models, enhance the effectiveness of human-computer interactions, or improve personal communication skills. However, quantifying engagement is challenging, since it is subjective and lacks a "gold standard". This paper proposes PMIScore, an efficient unsupervised approach to quantify dialogue engagement. It uses pointwise mutual information (PMI), which is the probability of generating a response conditioning on the conversation history. Thus, PMIScore offers a clear interpretation of engagement. As directly computing PMI is intractable due to the complexity of dialogues, PMIScore learned it through a dual form of divergence. The algorithm includes generating positive and negative dialogue pairs, extracting embeddings by large language models (LLMs), and training a small neural network using a mutual information loss function. We validated PMIScore on both synthetic and real-world datasets. Our results demonstrate the effectiveness of PMIScore in PMI estimation and the reasonableness of the PMI metric itself.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
