探索开放域问答系统在最小编辑问题上的对比一致性
Exploring Contrast Consistency of Open-Domain Question Answering Systems on Minimally Edited Questions
AI总结:
针对开放域问答(OpenQA)中对比一致性未被探索的问题,通过结合人工标注与大型语言模型生成最小编辑问题构建对比集,发现DPR表现不佳,进而提出查询侧对比损失与数据增强方法有效提升了DPR的对比一致性且不牺牲标准测试集准确性。
摘要:
对比一致性,即模型在存在扰动的情况下做出一致正确预测的能力,是NLP中的一个重要方面。虽然在情感分析和阅读理解等任务中已有研究,但由于难以收集满足事实性要求的扰动问题,它在开放域问答(OpenQA)中仍未被探索。在这项工作中,我们收集了最小编辑的问题作为具有挑战性的对比集来评估OpenQA模型。我们的收集方法结合了人工标注和大型语言模型生成。我们发现广泛使用的密集段落检索器(DPR)在我们的对比集上表现不佳,尽管它能很好地拟合训练集并在标准测试集上表现具有竞争力。为了解决这个问题,我们引入了一种简单有效的查询侧对比损失,并在数据增强的帮助下改进DPR训练。我们在对比集上的实验表明,DPR的对比一致性得到了提高,同时没有牺牲其在标准测试集上的准确性。
英文摘要:
Contrast consistency, the ability of a model to make consistently correct predictions in the presence of perturbations, is an essential aspect in NLP. While studied in tasks such as sentiment analysis and reading comprehension, it remains unexplored in open-domain question answering (OpenQA) due to the difficulty of collecting perturbed questions that satisfy factuality requirements. In this work, we collect minimally edited questions as challenging contrast sets to evaluate OpenQA models. Our collection approach combines both human annotation and large language model generation. We find that the widely used dense passage retriever (DPR) performs poorly on our contrast sets, despite fitting the training set well and performing competitively on standard test sets. To address this issue, we introduce a simple and effective query-side contrastive loss with the aid of data augmentation to improve DPR training. Our experiments on the contrast sets demonstrate that DPR's contrast consistency is improved without sacrificing its accuracy on the standard test sets.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。