DiPair：万亿级文本匹配和配对建模的快速准确提取
DiPair: Fast and Accurate Distillation for Trillion-Scale Text Matching and Pair Modeling
摘要：
预先训练的模型，如BERT（Devlin等人，2018）已经主导了NLP / IR应用，如单句分类，文本对分类和问答。然而，部署这些模型在实际系统中是非常不平凡的，由于其过高的计算成本。对此的一种常见的补救方法是知识蒸馏（Hinton等人，2015），导致更快的推理。然而，正如我们在这里所展示的，现有的作品并没有针对处理文本对（或元组）进行优化。因此，它们要么不可扩展，要么表现出低于标准的性能。在这项工作中，我们提出了DiPair-一种新的框架，用于在文本对任务上提取快速准确的模型。再加上端到端的训练策略，DiPair既具有高度可扩展性，又提供了改进的质量-速度权衡。对学术和现实世界的电子商务基准进行的实证研究表明，所提出的方法的效率与加速超过350倍和最小的质量下降相对于交叉注意教师BERT模型。
英文摘要：
Pre-trained models like BERT (Devlin et al., 2018) have dominated NLP / IR applications such as single sentence classification, text pair classification, and question answering. However, deploying these models in real systems is highly non-trivial due to their exorbitant computational costs. A common remedy to this is knowledge distillation (Hinton et al., 2015), leading to faster inference. However -- as we show here -- existing works are not optimized for dealing with pairs (or tuples) of texts. Consequently, they are either not scalable or demonstrate subpar performance. In this work, we propose DiPair -- a novel framework for distilling fast and accurate models on text pair tasks. Coupled with an end-to-end training strategy, DiPair is both highly scalable and offers improved quality-speed tradeoffs. Empirical studies conducted on both academic and real-world e-commerce benchmarks demonstrate the efficacy of the proposed approach with speedups of over 350x and minimal quality drop relative to the cross-attention teacher BERT model.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
