arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2304.01016cs.CLcs.AIcs.IR
收藏

快速密集检索器消耗 KALE:面向非对称双编码器的训练后嵌入 Kullback–Leibler 对齐

Quick Dense Retrievers Consume KALE: Post Training Kullback Leibler Alignment of Embeddings for Asymmetrical dual encoders

the University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Walmart Labs(沃尔玛实验室)

Daniel Campos, Alessandro Magnani, ChengXiang Zhai

AI总结:

本文针对密集检索双编码器的推理延迟问题,提出训练后查询编码器剪枝与 KL 对齐方法 KALE,并结合非对称模型设计,在无需完全重训或重建索引的情况下实现 3 倍加速且性能超过 DistilBERT。

摘要:

本文研究如何通过在上下文编码器与查询编码器之间引入结构化压缩和模型规模非对称性,来改善基于语言模型的密集检索系统的推理延迟。首先,我们考察训练前与训练后压缩在 MSMARCO、Natural Questions、TriviaQA、SQUAD 和 SCIFACT 上的影响,发现密集检索中双编码器的非对称性能够提升推理效率。基于这一认识,我们提出 Kullback Leibler Alignment of Embeddings(KALE,嵌入的 Kullback–Leibler 对齐),这是一种高效且准确的方法,通过在训练后剪枝并对齐查询编码器来提高密集检索方法的推理效率。具体而言,KALE 扩展了双编码器训练之后的传统知识蒸馏,使查询编码器能够得到有效压缩,而无需完全重新训练或生成索引。借助 KALE 和非对称训练,我们能够生成推理速度快 3 倍、同时性能超过 DistilBERT 的模型。

英文摘要:

In this paper, we consider the problem of improving the inference latency of language model-based dense retrieval systems by introducing structural compression and model size asymmetry between the context and query encoders. First, we investigate the impact of pre and post-training compression on the MSMARCO, Natural Questions, TriviaQA, SQUAD, and SCIFACT, finding that asymmetry in the dual encoders in dense retrieval can lead to improved inference efficiency. Knowing this, we introduce Kullback Leibler Alignment of Embeddings (KALE), an efficient and accurate method for increasing the inference efficiency of dense retrieval methods by pruning and aligning the query encoder after training. Specifically, KALE extends traditional Knowledge Distillation after bi-encoder training, allowing for effective query encoder compression without full retraining or index generation. Using KALE and asymmetric training, we can generate models which exceed the performance of DistilBERT despite having 3x faster inference.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑