查询漂移补偿：实现检索嵌入模型持续学习中的兼容性
Query Drift Compensation: Enabling Compatibility in Continual Learning of Retrieval Embedding Models
AI总结：
针对检索嵌入模型持续学习中模型更新与旧索引语料不兼容、需重索引的问题，提出查询漂移补偿方法，结合嵌入蒸馏将新查询嵌入投影到旧空间，无需重索引即可提升性能并减轻遗忘。
摘要：
文本嵌入模型可实现语义搜索，通过高效的信息检索（IR）为检索增强生成等多种NLP应用提供支持。然而，文本嵌入模型通常在训练数据静态的场景下被研究，这限制了其在随时间推移会出现新训练数据的动态场景中的应用。信息检索方法通常将大规模文档语料编码为低维嵌入，并将其存储在数据库索引中。检索过程中，系统会对语料执行语义搜索，并返回嵌入与查询嵌入最相似的文档。当用新训练数据更新嵌入模型时，由于生成语料嵌入的模型已发生变化，存在不兼容问题，直接使用已建立索引的语料的效果并不理想。虽然用更新后的模型重新索引旧语料文档可以实现兼容性，但这需要高得多的计算量和时间成本。因此，研究如何无需重新索引就能继续有效使用已建索引的语料至关重要。在本研究中，我们构建了一个包含大规模数据集的持续学习基准，在每个任务中用新数据集的查询-文档对持续训练稠密检索嵌入模型，观察到由于嵌入发生显著漂移，模型在旧任务上出现遗忘。我们对查询和文档嵌入均采用嵌入蒸馏以维持稳定性，并提出了一种新颖的检索阶段查询漂移补偿方法，将新模型的查询嵌入投影到旧嵌入空间中。这使得模型能够与使用旧模型提取的已索引语料嵌入兼容，从而减轻遗忘。我们的实验表明，所提出的方法在无需任何重新索引的情况下显著提升了性能。代码可在https://github.com/dipamgoswami/QDC获取。
英文摘要：
Text embedding models enable semantic search, powering several NLP applications like Retrieval Augmented Generation by efficient information retrieval (IR). However, text embedding models are commonly studied in scenarios where the training data is static, thus limiting its applications to dynamic scenarios where new training data emerges over time. IR methods generally encode a huge corpus of documents to low-dimensional embeddings and store them in a database index. During retrieval, a semantic search over the corpus is performed and the document whose embedding is most similar to the query embedding is returned. When updating an embedding model with new training data, using the already indexed corpus is suboptimal due to the non-compatibility issue, since the model which was used to obtain the embeddings of the corpus has changed. While re-indexing of old corpus documents using the updated model enables compatibility, it requires much higher computation and time. Thus, it is critical to study how the already indexed corpus can still be effectively used without the need of re-indexing. In this work, we establish a continual learning benchmark with large-scale datasets and continually train dense retrieval embedding models on query-document pairs from new datasets in each task and observe forgetting on old tasks due to significant drift of embeddings. We employ embedding distillation on both query and document embeddings to maintain stability and propose a novel query drift compensation method during retrieval to project new model query embeddings to the old embedding space. This enables compatibility with previously indexed corpus embeddings extracted using the old model and thus reduces the forgetting. We show that the proposed method significantly improves performance without any re-indexing. Code is available at https://github.com/dipamgoswami/QDC.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
