arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2311.07930cs.CL
收藏

一切皆相对!——一种用于提升零样本相关性预测的合成查询生成方法

It's All Relative! -- A Synthetic Query Generation Approach for Improving Zero-Shot Relevance Prediction

Google Research(谷歌研究院)

Aditi Chaudhary, Karthik Raman, Michael Bendersky

AI总结:

提出一种新的合成查询生成方法,通过同时生成不同标签的查询,并利用相对性简化推理,在七个IR数据集上提升零样本相关性预测性能。

摘要:

大型语言模型(LLM)的最新发展表明,通过仅提供8个演示示例进行提示,它们能够生成合成的查询-文档对。这使得构建更好的信息检索(IR)模型成为可能,尤其是在没有现成训练数据的任务中。通常,此类合成查询生成(QGen)方法以输入上下文(例如文本文档)为条件,生成与该上下文相关的查询,或者额外以相关性标签(例如相关与不相关)为条件,生成跨相关性桶的查询。然而,我们发现这类QGen方法并非最优,因为它们要求模型从少量示例中推理出期望的标签和输入。在这项工作中,我们提出通过同时为不同标签生成查询来减轻LLM的这一负担。我们假设,与其要求模型在给定输入上下文的情况下生成例如不相关的查询,不如要求模型生成相对于相关查询的不相关查询,这对模型而言是更简单的推理任务设置。在七个IR数据集上的广泛实验表明,以这种方式生成的合成查询能够带来更好的下游性能,这表明生成的查询确实具有更高的质量。

英文摘要:

Recent developments in large language models (LLMs) have shown promise in their ability to generate synthetic query-document pairs by prompting with as few as 8 demonstrations. This has enabled building better IR models, especially for tasks with no training data readily available. Typically, such synthetic query generation (QGen) approaches condition on an input context (e.g. a text document) and generate a query relevant to that context, or condition the QGen model additionally on the relevance label (e.g. relevant vs irrelevant) to generate queries across relevance buckets. However, we find that such QGen approaches are sub-optimal as they require the model to reason about the desired label and the input from a handful of examples. In this work, we propose to reduce this burden of LLMs by generating queries simultaneously for different labels. We hypothesize that instead of asking the model to generate, say, an irrelevant query given an input context, asking the model to generate an irrelevant query relative to a relevant query is a much simpler task setup for the model to reason about. Extensive experimentation across seven IR datasets shows that synthetic queries generated in such a fashion translates to a better downstream performance, suggesting that the generated queries are indeed of higher quality.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑