arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2004.14269cs.IR
收藏

开放领域答案重新排名的培训课程

Training Curricula for Open Domain Answer Re-Ranking

Sean MacAvaney, Franco Maria Nardini, Raffaele Perego, Nicola Tonellotto, Nazli Goharian, Ophir Frieder

备注：
Accepted at SIGIR 2020 (long)
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

摘要：

在面向精度的任务中，如答案排序，对许多相关答案进行高度排序比检索所有相关答案更重要。因此，一个好的排名策略是首先学习如何识别最简单的正确答案（即，向具有通常指示相关性的特征的答案分配高等级分数，而向具有不指示相关性的特征的答案分配低等级分数），在结合更复杂的逻辑以处理困难的情况（例如，语义匹配或推理）。在这项工作中，我们将这一想法应用于训练神经答案排名使用课程学习。我们提出了几种算法来估计给定训练样本的难度。我们表明，所提出的算法可以用来建立一个训练课程，在训练过程的早期降低困难样本的权重。随着训练过程的进行，我们的方法逐渐转变为对所有样本进行平等加权，而不管难度如何。我们提出了一个全面的评估，我们提出的想法在三个答案排名数据集。结果表明，我们的方法导致两个领先的神经排名架构，即BERT和ConvKNRM，使用逐点和成对损失的优越性能。当应用于基于BERT的排名器时，我们的方法在MRR上提高了4%，在P@1上提高了9%（与没有课程训练的模型相比）。这导致模型可以实现与更昂贵的最先进技术相当的性能。

英文摘要：

In precision-oriented tasks like answer ranking, it is more important to rank many relevant answers highly than to retrieve all relevant answers. It follows that a good ranking strategy would be to learn how to identify the easiest correct answers first (i.e., assign a high ranking score to answers that have characteristics that usually indicate relevance, and a low ranking score to those with characteristics that do not), before incorporating more complex logic to handle difficult cases (e.g., semantic matching or reasoning). In this work, we apply this idea to the training of neural answer rankers using curriculum learning. We propose several heuristics to estimate the difficulty of a given training sample. We show that the proposed heuristics can be used to build a training curriculum that down-weights difficult samples early in the training process. As the training process progresses, our approach gradually shifts to weighting all samples equally, regardless of difficulty. We present a comprehensive evaluation of our proposed idea on three answer ranking datasets. Results show that our approach leads to superior performance of two leading neural ranking architectures, namely BERT and ConvKNRM, using both pointwise and pairwise losses. When applied to a BERT-based ranker, our method yields up to a 4% improvement in MRR and a 9% improvement in P@1 (compared to the model trained without a curriculum). This results in models that can achieve comparable performance to more expensive state-of-the-art techniques.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑