SpaceNLI：评估空间中推理预测的一致性
SpaceNLI: Evaluating the Consistency of Predicting Inferences in Space
AI总结：
本文提出空间推理数据集 SpaceNLI，并引入模式准确率指标，发现现有 NLI 系统在空间推理上表现中等且缺乏模式一致性，其中非投射性空间推理最具挑战性。
摘要：
虽然许多自然语言推理（NLI）数据集针对特定的语义现象，例如否定、时态与体、单调性和预设，但据我们所知，目前还没有涉及多种空间表达和推理类型的 NLI 数据集。我们通过半自动地创建一个用于空间推理的 NLI 数据集来填补这一空白，该数据集称为 SpaceNLI。数据样本是从一组精心整理的推理模式中自动生成的，其中这些模式由专家标注了推理标签。我们在 SpaceNLI 上测试了多个最先进的 NLI 系统，以衡量该数据集的复杂性以及系统进行空间推理的能力。此外，我们引入了模式准确率（Pattern Accuracy），并认为与准确率相比，它是评估系统在基于模式生成的数据样本上性能的更可靠、更严格的度量。基于评估结果，我们发现这些系统在空间 NLI 问题上取得了中等水平的结果，但在每个推理模式上缺乏一致性。结果还表明，非投射性空间推理（尤其是由于“between”介词引起的推理）是最具挑战性的。
英文摘要：
While many natural language inference (NLI) datasets target certain semantic phenomena, e.g., negation, tense & aspect, monotonicity, and presupposition, to the best of our knowledge, there is no NLI dataset that involves diverse types of spatial expressions and reasoning. We fill this gap by semi-automatically creating an NLI dataset for spatial reasoning, called SpaceNLI. The data samples are automatically generated from a curated set of reasoning patterns, where the patterns are annotated with inference labels by experts. We test several SOTA NLI systems on SpaceNLI to gauge the complexity of the dataset and the system's capacity for spatial reasoning. Moreover, we introduce a Pattern Accuracy and argue that it is a more reliable and stricter measure than the accuracy for evaluating a system's performance on pattern-based generated data samples. Based on the evaluation results we find that the systems obtain moderate results on the spatial NLI problems but lack consistency per inference pattern. The results also reveal that non-projective spatial inferences (especially due to the "between" preposition) are the most challenging ones.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
