口语对话系统的分布性评估指标
Distributional Metrics for Evaluating Spoken Conversational Systems
- KTH Royal Institute of Technology（KTH皇家理工学院）
- NICT（日本信息通信研究机构）
- Charles University（查理大学）
- Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）
- University of Edinburgh（爱丁堡大学）
机构由 AI 辅助整理，请以论文原文为准。
中文总结 AI 辅助
本文提出对话分布得分（CDS），通过可解释特征比较对话行为分布，以人类对话为参考，评估口语对话系统，并验证其排名与听者偏好的一致性。
中文摘要 AI 辅助
评估对话系统是一个困难且尚未解决的问题。我们引入了对话分布得分（CDS），该得分以人类对话为参考，比较对话行为的分布。CDS通过八个可解释的特征以及一个独立的双特征语义基线来描述语速、音节节奏和话轮交互。我们将对话与两种参考尺度进行比较：一种基于人类对话中的对话成功度，另一种对比人类与合成对话。利用来自域外目标导向对话的听者判断，我们考察了系统排名、对话之间的偏好以及排名的稳定性。复合CDS恢复了六项听者系统比较中的五项，而单个特征显示出与听者对话间偏好的强相关性。我们考察了排名稳定所需的分钟数和对话数。这些发现支持将分布性比较作为特定交互指标的补充，用于评估对话和对话模型，同时展示了其可解释的价值。
英文摘要
Evaluating conversational systems is a difficult and unresolved problem. We introduce the Conversational Distribution Score (CDS), which compares distributions of conversational behaviour using human conversations as a reference. CDS describes speech rate, syllabic rhythm, and turn interaction through eight interpretable features plus a separate two-feature semantic baseline. We compare conversations with two reference scales: one based on conversational success within human dialogue and another contrasting human and synthetic dialogue. Using listener judgments from out-of-domain goal--oriented dialogues, we examine system ranking, preferences between conversations, and ranking stability. Composite CDS recovers five of six listener system comparisons while individual features show strong correlation with listener preferences between conversations. We examine how many minutes and conversations are required before rankings stabilize. These findings support distributional comparisons as a complement to specific interactional metrics to evaluate conversations and conversational models while showing their interpretable value.
