大型神经模型（如 Transformer）在信息检索（IR）中取得了最先进的性能。本文旨在改进蒸馏方法，为此类模型在实际中的资源高效部署铺平道路。受我们对 IR 模型师生泛化差距的理论分析启发，我们提出了一种新颖的蒸馏方法，该方法利用大型教师模型学习到的查询与文档之间的相对几何关系。与现有的基于教师分数的蒸馏方法不同，我们提出的方法采用嵌入匹配任务来提供更强的信号，以对齐教师模型和学生模型的表示。此外，它还利用查询生成来探索数据流形，以减少在训练数据稀疏的区域中学生模型与教师模型之间的差异。进一步地，我们的分析还推动了新颖的非对称学生模型架构的设计，在不增加在线推理成本的情况下实现更好的嵌入对齐。在 MSMARCO 等标准基准上，我们表明我们的方法成功地从双编码器（DE）和交叉编码器（CE）教师模型蒸馏出 1/10 规模的非对称学生模型，并能保留教师模型 95-97% 的性能。
英文摘要：
Large neural models (such as Transformers) achieve state-of-the-art performance for information retrieval (IR). In this paper, we aim to improve distillation methods that pave the way for the resource-efficient deployment of such models in practice. Inspired by our theoretical analysis of the teacher-student generalization gap for IR models, we propose a novel distillation approach that leverages the relative geometry among queries and documents learned by the large teacher model. Unlike existing teacher score-based distillation methods, our proposed approach employs embedding matching tasks to provide a stronger signal to align the representations of the teacher and student models. In addition, it utilizes query generation to explore the data manifold to reduce the discrepancies between the student and the teacher where training data is sparse. Furthermore, our analysis also motivates novel asymmetric architectures for student models which realizes better embedding alignment without increasing online inference cost. On standard benchmarks like MSMARCO, we show that our approach successfully distills from both dual-encoder (DE) and cross-encoder (CE) teacher models to 1/10th size asymmetric students that can retain 95-97% of the teacher performance.
Momentum Posterior Regularization for Multi-hop Dense Retrieval
State Key Laboratory for Novel Software Technology, Nanjing University（南京大学现代软件技术国家重点实验室）；School of Computer Science, Nanjing University（南京大学计算机科学学院）；School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）
LIX: Implicitly Infusing Spatial Geometric Prior Knowledge into Visual Semantic Segmentation for Autonomous Driving
College of Electronics & Information Engineering, Tongji University（同济大学电子与信息工程学院）；University of Thessaloniki（塞萨洛尼基大学）；Shanghai Institute of Intelligent Science and Technology（上海智能科学技术研究院）；Shanghai Research Institute for Intelligent Autonomous Systems（上海智能自主系统研究院）；Tongji University（同济大学）；Xi’an Jiaotong University（西安交通大学）
Cross-Modal Fusion Distillation for Fine-Grained Sketch-Based Image Retrieval
University of Exeter（埃克塞特大学）；University of Tübingen（图宾根大学）；MPI for Informatics（马克斯·普朗克信息学研究所）；MPI for Intelligent Systems（马克斯·普朗克智能系统研究所）；University of Surrey（萨里大学）
Ontology-Based Concept Distillation for Radiology Report Retrieval and Labeling
Department of Artificial Intelligence in Biomedical Engineering, Friedrich-Alexander-Universität Erlangen-Nürnberg（生物医学工程人工智能系，埃尔兰根-纽伦堡弗里德里希-亚历山大大学）；Department of Computing, Imperial College London（计算系，伦敦帝国理工学院）
BiXSE: Improving Dense Retrieval via Probabilistic Graded Relevance Distillation
Université de Montéal（蒙特阿尔大学）；Mila - Quebec AI Institute（魁北克AI研究院）；McGill University（麦吉尔大学）；Apple MLR（苹果机器学习研究部）；IVADO, Canada（加拿大IVADO研究所）；CIFAR AI Chair（CIFAR人工智能主席）；ServiceNow Research（ServiceNow研究）
DRAG: Distilling RAG for SLMs from LLMs to Transfer Knowledge and Mitigate Hallucination via Evidence and Graph-based Distillation
VILA Lab（VILA实验室）；Mohamed bin Zayed University of AI（莫扎伊德·本·泽德人工智能大学）；McGill University（麦吉尔大学）；National University of Science and Technology（国家科学与技术大学）
