arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2202.10408cs.CL
收藏

溯因自然语言推理的简单性能预测

Embarrassingly Simple Performance Prediction for Abductive Natural Language Inference

Institut für Maschinelle Sprachverarbeitung(机器语言处理研究所);University of Stuttgart(斯图加特大学)

Emīls Kadiķis, Vaibhav Srivastav, Roman Klinger

摘要:

溯因自然语言推理(αnli)的任务是决定哪个假设是对一组观察结果的更可能的解释,这是一种特别困难的NLI类型。它不仅需要确定因果关系,还需要常识来评估解释的合理性。所有最近的竞争系统都建立在上下文表示的基础上,并利用Transformer架构来学习NLI模型。当有人面临特定的NLI任务时,他们需要选择可用的最佳模型。这是一项耗费时间和资源的工作。为了解决这个实际问题,我们提出了一个简单的方法来预测的性能,实际上没有微调的模型。我们通过测试预先训练的模型在αnli任务上的表现如何来做到这一点,当仅仅比较具有余弦相似性的句子嵌入与在这些嵌入之上训练分类器时所实现的性能时。我们发现,余弦相似性方法的准确性与皮尔逊相关系数为0.65的分类方法的准确性密切相关。由于相似性计算在给定数据集上的计算速度快了几个数量级(不到一分钟,而不是几小时),因此我们的方法可以在模型选择过程中节省大量时间。

英文摘要:

The task of abductive natural language inference (αnli), to decide which hypothesis is the more likely explanation for a set of observations, is a particularly difficult type of NLI. Instead of just determining a causal relationship, it requires common sense to also evaluate how reasonable an explanation is. All recent competitive systems build on top of contextualized representations and make use of transformer architectures for learning an NLI model. When somebody is faced with a particular NLI task, they need to select the best model that is available. This is a time-consuming and resource-intense endeavour. To solve this practical problem, we propose a simple method for predicting the performance without actually fine-tuning the model. We do this by testing how well the pre-trained models perform on the αnli task when just comparing sentence embeddings with cosine similarity to what the performance that is achieved when training a classifier on top of these embeddings. We show that the accuracy of the cosine similarity approach correlates strongly with the accuracy of the classification approach with a Pearson correlation coefficient of 0.65. Since the similarity computation is orders of magnitude faster to compute on a given dataset (less than a minute vs. hours), our method can lead to significant time savings in the process of model selection.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑