SETI：文本推理的系统性评估
SETI: Systematicity Evaluation of Textual Inference
AI总结：
提出SETI基准，通过三种NLI任务评估PLMs在文本推理中的系统性能力，发现模型在具备组合知识时表现良好，否则大幅下降，且少量关键知识可显著提升性能。
摘要：
我们提出了SETI（Systematicity Evaluation of Textual Inference，文本推理的系统性评估），这是一个新颖且全面的基准，旨在评估预训练语言模型（PLMs）在文本推理领域的系统性能力。具体而言，SETI提供了三种不同的NLI任务及相应的数据集，以评估推理过程中各种类型的系统性。为了解决这些任务，模型需要基于已知的原始成分执行组合推理。我们在六种广泛使用的PLMs上进行了SETI实验。结果表明，当掌握了如何组合原始成分的知识时，各种PLMs能够解决未见过的组合推理，且表现良好。然而，当模型不具备这一知识时，其能力受到极大限制（下降40-100个百分点）。此外，我们发现，一旦在极小样本中接触到关键组合知识，PLMs就能大幅提升。这些发现确立了SETI作为衡量PLMs在文本推理中实现系统性泛化未来进展的首个基准。
英文摘要：
We propose SETI (Systematicity Evaluation of Textual Inference), a novel and comprehensive benchmark designed for evaluating pre-trained language models (PLMs) for their systematicity capabilities in the domain of textual inference. Specifically, SETI offers three different NLI tasks and corresponding datasets to evaluate various types of systematicity in reasoning processes. In order to solve these tasks, models are required to perform compositional inference based on known primitive constituents. We conduct experiments of SETI on six widely used PLMs. Results show that various PLMs are able to solve unseen compositional inferences when having encountered the knowledge of how to combine primitives, with good performance. However, they are considerably limited when this knowledge is unknown to the model (40-100% points decrease). Furthermore, we find that PLMs can improve drastically once exposed to crucial compositional knowledge in minimalistic shots. These findings position SETI as the first benchmark for measuring the future progress of PLMs in achieving systematicity generalization in the textual inference.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
