Many information retrieval tasks require large labeled datasets for fine-tuning. However, such datasets are often unavailable, and their utility for real-world applications can diminish quickly due to domain shifts. To address this challenge, we develop and motivate a method for using large language models (LLMs) to generate large numbers of synthetic queries cheaply. The method begins by generating a small number of synthetic queries using an expensive LLM. After that, a much less expensive one is used to create large numbers of synthetic queries, which are used to fine-tune a family of reranker models. These rerankers are then distilled into a single efficient retriever for use in the target domain. We show that this technique boosts zero-shot accuracy in long-tail domains and achieves substantially lower latency than standard reranking methods.
Distillation versus Contrastive Learning: How to Train Your Rerankers
Kahlert School of Computing, University of Utah(犹他大学计算机学院);Scientific Computing and Imaging Institute, University of Utah(犹他大学科学计算与成像研究所);Capital One, Inc(Capital One公司);The University of Queensland(昆士兰大学)
Exploring the Limits of Model Compression in LLMs: A Knowledge Distillation Study on QA Tasks
Fraunhofer Institute for Intelligent Analysis and Information Systems(弗劳恩霍夫智能分析与信息系统研究所);University of Southern Denmark(南丹麦大学);Fraunhofer Institute for Applied Information Technology(弗劳恩霍夫应用信息科技研究所);University Hospital of Cologne(科隆大学医院)
Rationale-Guided Knowledge Distillation for Cross-Lingual Stance Detection
School of Foreign Studies, Hefei University of Technology(合肥工业大学外国语学院);School of Computer Science and Information Engineering, Hefei University of Technology(合肥工业大学计算机与信息工程学院);Hefei Institute of Technology(合肥学院)
Tevatron Meets Megatron: Expert-Parallel LLM Reranker Training on an Academic Budget
University of Utah(犹他大学);University of Waterloo(滑铁卢大学);The University of Queensland(昆士兰大学);Carnegie Mellon University(卡内基梅隆大学);University of Virginia(弗吉尼亚大学)
SQuaD-SQL: Efficient Text-to-SQL with Small Language Models via LLM-Guided Knowledge Distillation
The University of Liverpool(利物浦大学);University of Macau(澳门大学);University of Chinese Academy of Sciences(中国科学院大学);Tsinghua University(清华大学);Microsoft(微软);Shanghai University of International Business and Economics(上海国际经贸管理大学);Huizhou University(惠州市大学)
HarmAug: Effective Data Augmentation for Knowledge Distillation of Safety Guard Models
KAIST(韩国科学技术院);Theori;Université de Montréal(蒙特利尔大学);Mila – Québec AI Institute(米拉-魁北克人工智能研究所);Technische Hochschule Nürnberg Georg Simon Ohm(纽伦堡应用技术大学);CIFAR AI Chair(加拿大高级研究所人工智能主席项目)