arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2212.10467cs.CL
收藏

基于差分分析与解释的通用时序推理

Generic Temporal Reasoning with Differential Analysis and Explanation

University of Pennsylvania(宾夕法尼亚大学)

Yu Feng, Ben Zhou, Haoyu Wang, Helen Jin, Dan Roth

AI总结:

本文提出 TODAY 任务,通过时序差分分析和人类解释标注评估并提升时序推理模型的泛化能力,揭示现有模型依赖虚假信息的缺陷,并利用联合学习和噪声监督推动通用时序推理系统的发展。

摘要:

时序推理是预测事件对之间时序关系的任务。尽管时序推理模型在领域内基准上表现尚可,但由于现有数据集的局限性,我们对这些系统的泛化能力知之甚少。本文引入了一个名为 TODAY 的新任务,通过时序差分分析来弥补这一不足。顾名思义,该任务评估系统能否正确理解增量变化的影响。具体而言,TODAY 对给定事件对引入细微的上下文变化,并要求系统判断这种细微的上下文变化会如何影响相关的时序关系分布。为促进学习,TODAY 还标注了人类解释。我们表明,现有模型(包括 GPT-3.5)在 TODAY 上表现降至随机猜测水平,说明它们在时序预测中严重依赖虚假信息而非进行正确推理。另一方面,我们证明 TODAY 的监督方式和解释标注可用于联合学习,鼓励模型在训练过程中使用更合适的信号,从而在多个基准上取得更优表现。TODAY 还可用于训练模型从 GPT-3.5 等噪声来源中获取附带监督信号,从而进一步推动通用时序推理系统的目标实现。

英文摘要:

Temporal reasoning is the task of predicting temporal relations of event pairs. While temporal reasoning models can perform reasonably well on in-domain benchmarks, we have little idea of these systems' generalizability due to existing datasets' limitations. In this work, we introduce a novel task named TODAY that bridges this gap with temporal differential analysis, which as the name suggests, evaluates whether systems can correctly understand the effect of incremental changes. Specifically, TODAY introduces slight contextual changes for given event pairs, and systems are asked to tell how this subtle contextual change would affect relevant temporal relation distributions. To facilitate learning, TODAY also annotates human explanations. We show that existing models, including GPT-3.5, drop to random guessing on TODAY, suggesting that they heavily rely on spurious information rather than proper reasoning for temporal predictions. On the other hand, we show that TODAY's supervision style and explanation annotations can be used in joint learning, encouraging models to use more appropriate signals during training and thus outperform across several benchmarks. TODAY can also be used to train models to solicit incidental supervision from noisy sources such as GPT-3.5, thus moving us more toward the goal of generic temporal reasoning systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑