Natural Language Inference (NLI) or Recognizing Textual Entailment (RTE) is the task of predicting the entailment relation between a pair of sentences (premise and hypothesis). This task has been described as a valuable testing ground for the development of semantic representations, and is a key component in natural language understanding evaluation benchmarks. Models that understand entailment should encode both, the premise and the hypothesis. However, experiments by Poliak et al. revealed a strong preference of these models towards patterns observed only in the hypothesis, based on a 10 dataset comparison. Their results indicated the existence of statistical irregularities present in the hypothesis that bias the model into performing competitively with the state of the art. While recast datasets provide large scale generation of NLI instances due to minimal human intervention, the papers that generate them do not provide fine-grained analysis of the potential statistical patterns that can bias NLI models. In this work, we analyze hypothesis-only models trained on one of the recast datasets provided in Poliak et al. for word-level patterns. Our results indicate the existence of potential lexical biases that could contribute to inflating the model performance.
Exploring Classic and Neural Lexical Translation Models for Information Retrieval: Interpretability, Effectiveness, and Efficiency Benefits
Bosch Center for Artificial Intelligence（博世人工智能中心）；Carnegie Mellon University（卡内基梅隆大学）
我们研究了词汇翻译模型（IBM Model 1）在英语文本检索中的实用性，特别是其端到端训练的神经变体。我们使用神经模型1作为聚合器层，应用于上下文无关或上下文化的查询/文档嵌入。这种设计神经排序系统的新方法在有效性、效率和可解释性方面都有好处。具体来说，我们表明，在基于BERT的上下文嵌入之上添加可解释的神经模型1层（1）不会降低准确性和/或效率;（2）可以克服现有BERT模型的最大序列长度限制。与上下文无关的神经模型1不如基于BERT的排名模型有效，但它可以在CPU上高效运行（无需对大型张量进行昂贵的索引时预计算或查询时操作）。使用模型1，我们在2020年底的MS MARCO文档排序排行榜上产生了最佳神经和非神经运行。
Topic Aware Probing: From Sentence Length Prediction to Idiom Identification how reliant are Neural Language Models on Topic?
ADAPT Research Centre（ADAPT研究中心）；Technological University Dublin（都柏林理工大学）；School of Computer Science and Statistics（计算机科学与统计学院）；Trinity College Dublin（都柏林圣三一学院）
提出主题感知探测方法，通过 BERT 与 RoBERTa 在句长预测到习语识别等任务上的表现，揭示模型对主题信息的依赖。
