DEPT：用于统一查询扩展与检索的文档嵌入保留调优

中文摘要 AI 辅助

大型语言模型（LLM）既可以扩展语义不明确的查询，又能将文本编码为稠密表示，这提示可以构建一个统一模型来完成查询扩展与检索任务。现有系统通常依赖提示式扩展、独立训练的模块或分阶段优化，导致生成的扩展结果仅与评判它们的检索损失间接对齐。我们端到端训练单个仅解码器的LLM，同一模型既生成扩展内容，又对扩展后的查询和候选文档进行编码。这种统一设置会产生移动目标问题：检索监督应改进查询侧的扩展，但相同的更新也会改变作为检索目标的文档嵌入。我们提出文档嵌入保留调优（DEPT），该方法在允许检索梯度通过直通解码传递到生成器的同时，使调优后的文档嵌入接近缓存的初始嵌入。DEPT将查询-文档的联合移动转化为针对近似稳定、白化的文档嵌入的查询侧适配，支持索引复用和在线难负样本挖掘。在BE基准的5个数据集上，使用Qwen3-4B-Instruct-2507和LLaMA-3.2-3B-Instruct进行的实验表明，DEPT相比无训练、独立训练和分阶段统一基线，提升了平均检索质量，而消融实验则分离了保留、白化、端到端扩展训练和在线负样本的影响。代码可在this https URL获取。

英文摘要

Large language models (LLMs) can both expand underspecified queries and encode text as dense representations, suggesting a unified model for query expansion and retrieval. Existing systems usually rely on prompted expansions, independently trained modules, or staged optimization, leaving generated expansions only indirectly aligned with the retrieval loss that judges them. We train a single decoder-only LLM end to end, where the same model generates the expansion and encodes both the expanded query and candidate documents. This unified setting creates a moving-target problem: retrieval supervision should improve query-side expansion, but the same update also shifts the document embeddings that serve as retrieval targets. We introduce Document Embedding Preservation Tuning (DEPT), which keeps tuned document embeddings close to cached initial embeddings while allowing retrieval gradients to pass through straight-through decoding into the generator. DEPT converts joint query--document movement into query-side adaptation against approximately stable, whitened document embeddings that support index reuse and online hard-negative mining. Experiments with Qwen3-4B-Instruct-2507 and LLaMA-3.2-3B-Instruct on five datasets in BEIR benchmark show that DEPT improves average retrieval quality over training-free, independently trained, and staged unified baselines, while ablations isolate the effects of preservation, whitening, end-to-end expansion training, and online negatives. Code is available at https://github.com/ILSparkle/DEPT.

发表机构

• Beihang University（北京航空航天大学）

机构由 AI 辅助整理，请以论文原文为准。