你的目标导向对话模型真的表现得很好吗？系统评价的实证分析
Is Your Goal-Oriented Dialog Model Performing Really Well? Empirical Analysis of System-wise Evaluation
摘要：
人们对开发面向目标的对话系统越来越感兴趣，该系统通过多轮对话为用户完成复杂任务提供服务。虽然设计了许多方法来评估和提高单个对话组件的性能，但缺乏对不同组件如何对对话系统的整体性能做出贡献的全面实证研究。在本文中，我们进行了系统的评估，并提出了不同类型的对话系统，这是由不同的模块在不同的设置进行了实证分析。我们的研究结果表明，（1）使用不同组件级别的细粒度监督信号训练的管道对话系统通常比使用粗粒度标签训练的联合或端到端模型的系统获得更好的性能，（2）组件方面的单轮评估结果并不总是与对话系统的整体性能一致，（3）尽管模拟器和人类用户之间存在差异，但模拟评估仍然是昂贵的人类评估的有效替代方案，特别是在开发的早期阶段。
英文摘要：
There is a growing interest in developing goal-oriented dialog systems which serve users in accomplishing complex tasks through multi-turn conversations. Although many methods are devised to evaluate and improve the performance of individual dialog components, there is a lack of comprehensive empirical study on how different components contribute to the overall performance of a dialog system. In this paper, we perform a system-wise evaluation and present an empirical analysis on different types of dialog systems which are composed of different modules in different settings. Our results show that (1) a pipeline dialog system trained using fine-grained supervision signals at different component levels often obtains better performance than the systems that use joint or end-to-end models trained on coarse-grained labels, (2) component-wise, single-turn evaluation results are not always consistent with the overall performance of a dialog system, and (3) despite the discrepancy between simulators and human users, simulated evaluation is still a valid alternative to the costly human evaluation especially in the early stage of development.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
