正确的错误的理由：诊断自然语言推理中的句法启发式
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
摘要：
机器学习系统可以通过依赖于对频繁示例类型有效但在更具挑战性的情况下会崩溃的算法，在给定的测试集上获得良好的分数。我们在自然语言推理（NLI）中研究这个问题，确定一个句子是否包含另一个句子。我们假设，统计NLI模型可能会采用三种易犯错误的句法启发式：词汇重叠启发式，子序列启发式和成分启发式。为了确定模型是否采用了这些启发式，我们引入了一个受控的评估集，称为HANS（启发式分析NLI系统），其中包含许多例子，其中的启发式失败。我们发现，在MNLI上训练的模型，包括最先进的模型BERT，在HANS上的表现非常差，这表明它们确实采用了这些算法。我们的结论是，NLI系统有很大的改进空间，HANS数据集可以激励和衡量这一领域的进展
英文摘要：
A machine learning system can score well on a given test set by relying on heuristics that are effective for frequent example types but break down in more challenging cases. We study this issue within natural language inference (NLI), the task of determining whether one sentence entails another. We hypothesize that statistical NLI models may adopt three fallible syntactic heuristics: the lexical overlap heuristic, the subsequence heuristic, and the constituent heuristic. To determine whether models have adopted these heuristics, we introduce a controlled evaluation set called HANS (Heuristic Analysis for NLI Systems), which contains many examples where the heuristics fail. We find that models trained on MNLI, including BERT, a state-of-the-art model, perform very poorly on HANS, suggesting that they have indeed adopted these heuristics. We conclude that there is substantial room for improvement in NLI systems, and that the HANS dataset can motivate and measure progress in this area
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
