arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.07555cs.CL
收藏

O1 Embedder:让检索模型在行动前思考

O1 Embedder: Let Retrievers Think Before Action

University of Science and Technology of China(中国科学技术大学)

Ruiran Yan, Zheng Liu, Defu Lian

AI总结:

受OpenAI O1等推理模型启发,提出O1 Embedder方法,让检索模型在检索前生成思考;通过LLM专家生成初始思考+检索委员会优化的训练信号,结合行为克隆与对比学习训练,在12个数据集上提升检索准确性与泛化性。

摘要:

大型语言模型(LLMs)日益增强的能力彻底改变了人们获取和利用信息的方式。值得注意的是,LLMs擅长执行细粒度的数据表示,这有助于精确的信息检索。它们还能基于外部参考生成高质量答案,从而产生有用的知识。最近推出的推理模型(如OpenAI O1和DeepSeek R1)标志着又一次飞跃,凸显了LLMs在提供最终答案前逐步思考的能力。这一突破显著提升了处理复杂任务的能力,例如编码和数学证明。受此进展启发,我们旨在为检索模型开发类似的能力,这类模型在应对多任务检索、零样本检索以及需要对复杂关系进行密集推理的任务等领域关键挑战方面具有巨大潜力。为实现这一目标,我们克服了两个技术难题:首先,设计了数据合成工作流,通过LLM专家生成初始思考,再利用检索委员会进行优化,为O1 Embedder创建训练信号;其次,优化了训练过程,使预训练模型能通过行为克隆联合微调以生成检索思考,并通过对比学习执行密集检索。我们通过全面实验对该方法进行评估,结果显示在12个流行数据集(涵盖域内和域外场景)上均取得了显著改进。这些结果凸显了O1 Embedder卓越的准确性和泛化性,为下一代信息检索(IR)基础模型的开发铺平了道路。

英文摘要:

The growing power of large language models (LLMs) has revolutionized how people access and utilize information. Notably, the LLMs excel at performing fine-grained data representation, which facilitates precise retrieval of information. They also generate high-quality answers based on external references, enabling the production of useful knowledge. The recent introduction of reasoning models, like OpenAI O1 and DeepSeek R1, marks another leap forward, highlighting LLMs' ability to think progressively before delivering final answers. This breakthrough significantly improves the ability to address complex tasks, e.g., coding and math proofs. Inspired by this progress, we aim to develop similar capabilities for retrieval models, which hold great promise for tackling critical challenges in the field, including multi-task retrieval, zero-shot retrieval, and tasks requiring intensive reasoning of complex relationships. With this motivation, we propose a novel approach called O1 Embedder, which generates useful thoughts for the input query before making retrieval for the target documents. To realize this objective, we conquer two technical difficulties. First, we design a data synthesis workflow, creating training signals for O1 Embedder by generating initial thoughts from an LLM-expert and subsequently refining them using a retrieval committee. Second, we optimize the training process, enabling a pre-trained model to be jointly fine-tuned to generate retrieval thoughts via behavior cloning and perform dense retrieval through contrastive learning. Our approach is evaluated by comprehensive experiments, where substantial improvements are achieved across 12 popular datasets, spanning both in-domain and out-of-domain scenarios. These results highlight O1 Embedder's remarkable accuracy and generalizability, paving the way for the development of next-generation IR foundation models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑