InRanker:面向零样本信息检索的蒸馏排序器
InRanker: Distilled Rankers for Zero-shot Information Retrieval
AI总结:
本文提出 InRanker 方法,通过生成合成域内数据和两阶段蒸馏将 monoT5-3B 的知识迁移到小型排序器,在零样本域外检索中显著提升 monoT5-60M 和 monoT5-220M 的有效性。
摘要:
尽管数十亿参数的神经排序器已成为最先进信息检索流程的常见组成部分,但由于推理所需的计算量巨大,它们很少在生产环境中使用。在这项工作中,我们提出了一种将大型排序器蒸馏为较小版本的新方法,重点关注域外有效性。我们引入了 InRanker,这是从 monoT5-3B 蒸馏而来的 monoT5 版本,在域外场景中具有更高的有效性。我们的关键见解是使用语言模型和重排序器尽可能多地生成合成的“域内”训练数据,即与检索时将看到的数据高度相似的数据。该流程包含两个蒸馏阶段,不需要额外的用户查询或人工标注:(1)在现有的有监督软教师标签上进行训练,以及(2)在使用大型语言模型生成的合成查询的教师软标签上进行训练。因此,像 monoT5-60M 和 monoT5-220M 这样的模型通过利用教师的知识提高了有效性,尽管它们分别小了 50 倍和 13 倍。模型和代码可在 https://github.com/unicamp-dl/InRanker 获取。
英文摘要:
Despite multi-billion parameter neural rankers being common components of state-of-the-art information retrieval pipelines, they are rarely used in production due to the enormous amount of compute required for inference. In this work, we propose a new method for distilling large rankers into their smaller versions focusing on out-of-domain effectiveness. We introduce InRanker, a version of monoT5 distilled from monoT5-3B with increased effectiveness on out-of-domain scenarios. Our key insight is to use language models and rerankers to generate as much as possible synthetic "in-domain" training data, i.e., data that closely resembles the data that will be seen at retrieval time. The pipeline consists of two distillation phases that do not require additional user queries or manual annotations: (1) training on existing supervised soft teacher labels, and (2) training on teacher soft labels for synthetic queries generated using a large language model. Consequently, models like monoT5-60M and monoT5-220M improved their effectiveness by using the teacher's knowledge, despite being 50x and 13x smaller, respectively. Models and code are available at https://github.com/unicamp-dl/InRanker.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。