残差轨迹蒸馏用于生成式检索

中文摘要 AI 辅助

生成式检索已成为一种通用的检索范式，它使用离散的语义ID（SIDs）来表示项目，并通过自回归的标识符生成来进行检索。当SIDs通过残差量化（RQ）构建时，标准的检索训练仅监督所选的码，并丢弃产生这些码的残差轨迹。然而，相同的硬码可能源于对竞争码字的不同偏好，而残差轨迹还包含有关后续量化决策的信息。因此，硬SID监督将不同的量化行为压缩为相同的目标，并使得索引期间可用的信息在检索训练中未被利用。我们引入了ResTD，一个残差轨迹蒸馏框架，将这种被丢弃的索引信息转移到检索训练中。将冻结的RQ索引器视为过程教师，它蒸馏残差诱导的码字偏好到SID解码状态中。这种监督恢复了硬分配所隐藏的区分，并允许较早的解码器状态在相应的SID后缀生成之前捕获有关后续量化决策的信息。通过这种方式，来自SID构建的更丰富信息被纳入检索学习，同时保留了原始的检索索引和推理过程。在多语言电子商务检索上的实验表明，与强基线和匹配的训练控制相比，取得了持续的改进。受控比较显示，残差衍生的目标优于测试的仅码本软目标。表示探针进一步表明，未来的码本偏好更容易从较早的解码器状态中恢复。ResTD还可以轻松扩展到生成式推荐。代码可在以下网址获取：此HTTPS URL。

英文摘要

Generative retrieval has emerged as a general retrieval paradigm, representing items with discrete Semantic IDs (SIDs) and retrieving them through autoregressive identifier generation. When SIDs are constructed with residual quantization (RQ), standard retrieval training supervises only the selected codes and discards the residual trajectories that produce them. The same hard code can nevertheless arise from different preferences over competing codewords, while the residual trajectory also contains information about subsequent quantization decisions. As a result, hard SID supervision collapses distinct quantization behaviors into identical targets and leaves information available during indexing unused in retrieval training. We introduce ResTD, a Residual Trajectory Distillation framework that transfers this discarded indexing information into retrieval training. Treating the frozen RQ indexer as a process teacher, it distills residual-induced codeword preferences into SID-decoding states. This supervision recovers distinctions hidden by hard assignments and allows earlier decoder states to capture information about subsequent quantization decisions before the corresponding SID suffix is generated. In this way, richer information from SID construction is incorporated into retrieval learning while preserving the original retrieval index and inference procedure. Experiments on multilingual e-commerce retrieval show consistent improvements over strong baselines and matched training controls. Controlled comparisons show that residual-derived targets outperform the tested codebook-only soft targets. Representation probes further show that future codebook preferences become more recoverable from earlier decoder states. ResTD can also be readily extended beyond retrieval to generative recommendation. Code is available at: https://github.com/Nevaeh7/iclr2027_ResTD.git.

发表机构

• Institute of Artificial Intelligence, Beihang University（北京航空航天大学人工智能研究院）
• Meituan（美团）

机构由 AI 辅助整理，请以论文原文为准。