arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.10487cs.IRcs.CL
收藏

生成式稠密检索:记忆可能成为负担

Generative Dense Retrieval: Memory Can Be a Burden

Beijing Institute of Technology(北京理工大学);Xiaohongshu Inc(小红书公司);Carnegie Mellon University(卡内基梅隆大学)

Peiwen Yuan, Xinglin Wang, Shaoxiong Feng, Boyuan Pan, Yiwei Li, Heda Wang, Xupeng Miao, Kan Li

AI总结:

针对生成式检索记忆细粒度信息不准、规模扩展时混淆及更新成本高的问题,本文提出GDR,结合GR簇间记忆匹配与DR无记忆簇内匹配,并辅以簇标识符构建和自适应负采样,在NQ上平均提升3.0 R@100。

摘要:

生成式检索(Generative Retrieval,GR)在给定查询时自回归解码相关文档标识符,已被证明在小规模语料库设置下表现良好。通过用模型参数记忆文档语料库,GR隐式实现了查询与文档之间的深度交互。然而,这种记忆机制面临三个缺点:(1)对文档细粒度特征的记忆准确性较差;(2)随着语料库规模增大,记忆混淆会加剧;(3)新文档的记忆更新成本巨大。为缓解这些问题,我们提出了生成式稠密检索(Generative Dense Retrieval,GDR)范式。具体而言,GDR首先利用有限的记忆容量实现从查询到相关文档簇的簇间匹配。随后引入来自稠密检索(Dense Retrieval,DR)的无记忆匹配机制,执行从簇到相关文档的细粒度簇内匹配。这种由粗到细的过程最大化了GR深度交互与DR可扩展性的优势。此外,我们设计了一种簇标识符构建策略以促进语料库记忆,并设计了簇自适应负采样策略以增强簇内映射能力。实证结果表明,GDR在多种设置下于NQ数据集上平均取得3.0 R@100的提升,并具有更好的可扩展性。

英文摘要:

Generative Retrieval (GR), autoregressively decoding relevant document identifiers given a query, has been shown to perform well under the setting of small-scale corpora. By memorizing the document corpus with model parameters, GR implicitly achieves deep interaction between query and document. However, such a memorizing mechanism faces three drawbacks: (1) Poor memory accuracy for fine-grained features of documents; (2) Memory confusion gets worse as the corpus size increases; (3) Huge memory update costs for new documents. To alleviate these problems, we propose the Generative Dense Retrieval (GDR) paradigm. Specifically, GDR first uses the limited memory volume to achieve inter-cluster matching from query to relevant document clusters. Memorizing-free matching mechanism from Dense Retrieval (DR) is then introduced to conduct fine-grained intra-cluster matching from clusters to relevant documents. The coarse-to-fine process maximizes the advantages of GR's deep interaction and DR's scalability. Besides, we design a cluster identifier constructing strategy to facilitate corpus memory and a cluster-adaptive negative sampling strategy to enhance the intra-cluster mapping ability. Empirical results show that GDR obtains an average of 3.0 R@100 improvement on NQ dataset under multiple settings and has better scalability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑