arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2109.04834cs.CLcs.AI
收藏

稳健多轮响应选择模型的评价数据集与策略

An Evaluation Dataset and Strategy for Building Robust Multi-turn Response Selection Model

Kakao Enterprise;SK Telecom(SK电信)

Kijong Han, Seojin Lee, Wooin Lee, Joosung Lee, Dong-hun Lee

摘要:

多回合反应选择模型最近在几个基准数据集中显示出与人类相当的性能。然而,在现实环境中,这些模型往往存在弱点,例如在没有全面了解上下文的情况下,严重基于表面模式做出错误的预测。例如,这些模型通常会给包含与上下文相关的几个关键字但使用不一致时态的错误响应候选人高分。在这项研究中,我们分析了开放域韩国多回合响应选择模型的弱点,并发布了一个对抗数据集来评估这些弱点。我们还提出了一个策略,在这种对抗性的环境中建立一个强大的模型。

英文摘要:

Multi-turn response selection models have recently shown comparable performance to humans in several benchmark datasets. However, in the real environment, these models often have weaknesses, such as making incorrect predictions based heavily on superficial patterns without a comprehensive understanding of the context. For example, these models often give a high score to the wrong response candidate containing several keywords related to the context but using the inconsistent tense. In this study, we analyze the weaknesses of the open-domain Korean Multi-turn response selection models and publish an adversarial dataset to evaluate these weaknesses. We also suggest a strategy to build a robust model in this adversarial environment.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑