arXiv 2605.072102026-05-27cs.IRcs.CL

DiffRetriever: 基于扩散语言模型的并行代表标记用于检索

DiffRetriever: Parallel Representative Tokens for Retrieval with Diffusion Language Models

The University of Queensland（昆士兰大学）；CSIRO（澳大利亚联邦科学与工业研究组织）

Shuai Wang, Yu Yin, Shengyao Zhuang, Bevan Koopman, Guido Zuccon

备注：: Updated analysis, ablation and benchmark with sota retrievers, indexing storage/latency ablation, isolating the effectiveness gain

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出DiffRetriever，利用扩散语言模型的掩码位置预测直接进行检索，支持单表示和多表示检索，在多个基准上优于现有方法。

摘要：

本文展示了扩散语言模型（DLM）如何作为有效且高效的检索器。现有的基于DLM的检索器（例如DiffEmbed）遵循BERT风格的编码，将每个查询或段落表示为单个平均池化向量。这忽略了DLM在双向注意力下通过掩码位置预测生成响应的能力，而这种能力可以提供更强的检索信号。我们提出DiffRetriever，它直接使用DLM原生的掩码位置预测进行检索。对于每个查询或段落，DiffRetriever附加一个或多个掩码位置，在单次前向传播中将输出用作检索表示。使用一个掩码位置时，单表示DiffRetriever在相同骨干网络上已经优于DiffEmbed。DiffRetriever还自然地扩展到多表示检索：DLM联合处理多个掩码位置，实现ColBERT风格的细粒度匹配，且编码延迟增加很小。在自回归LLM检索器中，相同的多表示策略需要顺序解码，因此产生更高的延迟。DiffRetriever在我们匹配的比较中获得了最强的整体效果，优于DiffEmbed、PromptReps和RepLLaMA。在训练数据上选择的掩码位置数量能够很好地跨数据集迁移，而每个查询的变化表明自适应分配仍有提升空间。代码可在https://github.com/ielab/diffretriever获取。

英文摘要：

This paper shows how diffusion language models (DLMs) can be used as effective and efficient retrievers. Existing DLM-based retrievers (e.g., DiffEmbed) follow BERT-style encoding, representing each query or passage as a single mean-pooled vector. This ignores how DLMs are trained to generate responses through masked-position prediction under bidirectional attention, a capability that can provide stronger retrieval signals. We propose DiffRetriever, which uses the DLM's native masked-position prediction directly for retrieval. For each query or passage, DiffRetriever appends one or more masked positions, using the outputs as retrieval representations in a single forward pass. With one masked position, single-representation DiffRetriever already improves over DiffEmbed on the same backbones. DiffRetriever also naturally extends to multi-representation retrieval: DLMs process multiple masked positions jointly, enabling ColBERT-style fine-grained matching with little additional encoding latency. In autoregressive LLM retrievers, the same multi-representation strategy requires sequential decoding and therefore incurs much higher latency. DiffRetriever obtains the strongest aggregate effectiveness within our matched comparison, outperforming DiffEmbed, PromptReps, and RepLLaMA. Masked-position counts selected on training data transfer well across datasets, while per-query variation suggests headroom for adaptive allocation. Code is available at https://github.com/ielab/diffretriever.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。