arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2106.05555cs.CL
收藏

BLEU的影子,成功的味道:MultiWOZ的案例

Shades of BLEU, Flavours of Success: The Case of MultiWOZ

Charles University(查理大学);Institute of Formal and Applied Linguistics(形式与应用语言学研究所)

Tomáš Nekvinda, Ondřej Dušek

摘要:

MultiWOZ数据集(Budzianowski等人,2018)经常用于对面向任务的对话系统的上下文响应能力进行基准测试。在这项工作中,我们确定了数据预处理和报告中的三个基于语料库的指标在这个数据集上使用的不一致,即,BLEU评分和Inform & Success率。我们指出了MultiWOZ基准测试的一些问题,如不满意的预处理,不足或指定的评估指标,或刚性数据库。我们在尽可能公平的设置中重新评估了7个端到端和6个策略优化模型,我们发现它们的报告分数不能直接比较。为了便于将来系统的比较,我们发布了独立的标准化评估脚本。最后,我们对未来基于语料库的基准测试工作提出了一些基本建议。

英文摘要:

The MultiWOZ dataset (Budzianowski et al.,2018) is frequently used for benchmarking context-to-response abilities of task-oriented dialogue systems. In this work, we identify inconsistencies in data preprocessing and reporting of three corpus-based metrics used on this dataset, i.e., BLEU score and Inform & Success rates. We point out a few problems of the MultiWOZ benchmark such as unsatisfactory preprocessing, insufficient or under-specified evaluation metrics, or rigid database. We re-evaluate 7 end-to-end and 6 policy optimization models in as-fair-as-possible setups, and we show that their reported scores cannot be directly compared. To facilitate comparison of future systems, we release our stand-alone standardized evaluation scripts. We also give basic recommendations for corpus-based benchmarking in future works.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑