arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1809.02719cs.CLcs.AI
收藏

如果我们简单地交换两个文本片段呢?一种简单有效的测试自然语言推理中混杂信号鲁棒性的方法

What If We Simply Swap the Two Text Fragments? A Straightforward yet Effective Way to Test the Robustness of Methods to Confounding Signals in Nature Language Inference Tasks

Carnegie Mellon University(卡内基梅隆大学);Southeast University(东南大学)

Haohan Wang, Da Sun, Eric P. Xing

摘要:

自然语言推理任务是确定一对自然语言句子之间推理关系的预测任务。随着NLI的日益普及,许多最先进的预测模型已经提出了令人印象深刻的性能。然而,一些作品已经注意到,在收集的NLI数据集,可能会导致这些模型的性能高估的统计违规行为,并提出了补救措施。在本文中,我们进一步调查的统计不规则性,我们称之为混杂因素,NLI数据集。由于相信一些NLI标签应该在交换操作下保留,我们提出了一种简单而有效的方法(交换两个文本片段)来评估自然缓解观察到的问题的NLI预测模型。此外,我们继续训练的预测模型与我们的交换方式,并提出使用不同的百分比的训练文本片段被交换的模型的评估性能的偏差来描述的预测模型的鲁棒性。我们的评估指标导致一些有趣的理解最近公布的NLI方法。最后,我们还将交换操作应用于NLI模型,以了解这种简单方法在为其他NLP传输任务训练通用句子嵌入时减轻混淆因素问题的有效性。

英文摘要:

Nature language inference (NLI) task is a predictive task of determining the inference relationship of a pair of natural language sentences. With the increasing popularity of NLI, many state-of-the-art predictive models have been proposed with impressive performances. However, several works have noticed the statistical irregularities in the collected NLI data set that may result in an over-estimated performance of these models and proposed remedies. In this paper, we further investigate the statistical irregularities, what we refer as confounding factors, of the NLI data sets. With the belief that some NLI labels should preserve under swapping operations, we propose a simple yet effective way (swapping the two text fragments) of evaluating the NLI predictive models that naturally mitigate the observed problems. Further, we continue to train the predictive models with our swapping manner and propose to use the deviation of the model's evaluation performances under different percentages of training text fragments to be swapped to describe the robustness of a predictive model. Our evaluation metrics leads to some interesting understandings of recent published NLI methods. Finally, we also apply the swapping operation on NLI models to see the effectiveness of this straightforward method in mitigating the confounding factor problems in training generic sentence embeddings for other NLP transfer tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑