面向任务型对话流的自动评估
Towards Automatic Evaluation of Task-Oriented Dialogue Flows
AI总结：
本文提出FuDGE（模糊对话图编辑距离）这一新颖指标，通过评估结构复杂性和对话数据表征覆盖度，解决任务型对话流缺乏标准评估方法的问题，有效提升对话设计的效率与自动化水平。
摘要：
任务型对话系统依赖于预定义的对话方案（对话流），这些方案通常表示为有向无环图。这些对话流可以人工设计，或从先前记录的对话中自动生成。由于领域专业知识的差异或对先前对话集的依赖不同，这些对话流可能呈现显著不同的图结构。尽管对话流很重要，但目前没有评估对话流质量的标准方法。我们提出了FuDGE（模糊对话图编辑距离），这是一种新颖的度量指标，通过评估对话流的结构复杂性和对对话数据的表征覆盖度来评估对话流。FuDGE衡量单个对话与对话流的匹配程度，进而衡量一组对话被该对话流整体表征的程度。通过对人工配置的对话流和自动化技术生成的对话流进行大量实验，我们证明了FuDGE及其评估框架的有效性。通过标准化和优化对话流，FuDGE使对话设计者和自动化技术能够实现更高的效率和自动化水平。
英文摘要：
Task-oriented dialogue systems rely on predefined conversation schemes (dialogue flows) often represented as directed acyclic graphs. These flows can be manually designed or automatically generated from previously recorded conversations. Due to variations in domain expertise or reliance on different sets of prior conversations, these dialogue flows can manifest in significantly different graph structures. Despite their importance, there is no standard method for evaluating the quality of dialogue flows. We introduce FuDGE (Fuzzy Dialogue-Graph Edit Distance), a novel metric that evaluates dialogue flows by assessing their structural complexity and representational coverage of the conversation data. FuDGE measures how well individual conversations align with a flow and, consequently, how well a set of conversations is represented by the flow overall. Through extensive experiments on manually configured flows and flows generated by automated techniques, we demonstrate the effectiveness of FuDGE and its evaluation framework. By standardizing and optimizing dialogue flows, FuDGE enables conversational designers and automated techniques to achieve higher levels of efficiency and automation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
