计算而非记忆:用于高效生成式检索的比特分解标识符
Computed, not Memorized: Bit-Factorized Identifiers for Efficient Generative Retrieval
AI总结:
本文提出一种比特分解的数字DocID设计,无需额外训练即可提升生成式检索对未见过文档的泛化能力,在多个基准上显著改善未见过文档检索性能并降低推理延迟。
摘要:
生成式检索以高效的端到端索引和检索著称。特别是,数字文档标识符(DocID)通常更高效,因为它们更短,且是基于小词汇表的固定长度序列,在束搜索期间需要更少的解码步骤。然而,它们对未见过的文档的泛化能力较差,必须在索引阶段训练中学习。在本文中,我们提出了一种用于生成式检索的数字DocID设计,该设计无需额外训练即可更好地适应未见过的文档。这种DocID设计继承了语言模型的知识:每个DocID是分解为比特的标记序列,每个标记的得分是其各比特得分的总和。与现有的数字DocID不同,现有DocID的标记仅通过索引训练期间记忆的嵌入来承载意义,而这里的每个比特都从模型的表示中衍生出自身的意义。因此,未见过文档的DocID仍然可以被评分,无需额外训练。在三个基准上的实验表明,我们的DocID设计在未见过文档的检索上显著提升性能,同时在已见过文档上保持可比的性能。在NQ320K和TriviaQA上,它在未见过文档的Hit@10指标上实现了超过20个百分点的提升,推理延迟比现有数字DocID设计低达7倍;在MS MARCO上,它的性能仅比权重更新方法低3个百分点,且无需额外训练。这些结果表明,合适的DocID设计可以大幅降低将生成式检索器适配到新文档时的重新训练成本。
英文摘要:
Generative retrieval is known for its efficient end-to-end indexing and retrieval. In particular, numeric Document Identifiers (DocIDs) are generally more efficient because they are shorter and generally fixed-length sequences over a small vocabulary, requiring fewer decoding steps during beam search. However, they tend to generalize poorly to unseen documents and must be learned during indexing-stage training. In this paper, we propose a numeric DocID design for generative retrieval that enables better adaptation to unseen documents without retraining. Our DocID design inherits the language model's knowledge: each DocID is a sequence of tokens factorized into bits, and each token is scored as the sum of its bits. Unlike existing numeric DocIDs, whose tokens carry meaning only through embeddings memorized during indexing training, each bit here derives its meaning from the model's representation. Therefore, the DocIDs of unseen documents remain scorable without retraining. Experiments on three benchmarks show that our DocID design substantially improves unseen-document retrieval while maintaining comparable performance on seen documents. It achieves over 20 percentage points higher Hit@10 on unseen documents and up to 7 times lower inference latency than existing numeric DocID designs on NQ320K and TriviaQA, while performing only 3 percentage points below weight-update methods on MS MARCO without additional training. These results show that an appropriate DocID design can heavily reduce retraining costs when adapting generative retrievers to new documents.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。