Large-scale pre-trained sequence-to-sequence models like BART and T5 achieve state-of-the-art performance on many generative NLP tasks. However, such models pose a great challenge in resource-constrained scenarios owing to their large memory requirements and high latency. To alleviate this issue, we propose to jointly distill and quantize the model, where knowledge is transferred from the full-precision teacher model to the quantized and distilled low-precision student model. Empirical analyses show that, despite the challenging nature of generative tasks, we were able to achieve a 16.5x model footprint compression ratio with little performance drop relative to the full-precision counterparts on multiple summarization and QA datasets. We further pushed the limit of compression ratio to 27.7x and presented the performance-efficiency trade-off for generative tasks using pre-trained models. To the best of our knowledge, this is the first work aiming to effectively distill and quantize sequence-to-sequence pre-trained models for language generation tasks.
最近的最先进的摘要方法利用大型预训练的Transformer模型。将这些模型提炼成更小的学生模型对于实际应用来说已经变得至关重要;然而,NLP文献提出了许多不同的提炼方法。最近的工作蒸馏BERT的分类和回归任务显示出强大的性能,使用直接知识蒸馏。或者,机器翻译从业者使用伪标记进行提取,其中小模型在较大模型的翻译上进行训练。第三种更简单的方法是“收缩和微调”(SFT),它通过将参数复制到较小的学生模型然后进行微调来避免任何显式蒸馏。我们比较了这三种方法对Pegasus和BART的蒸馏,当前和以前的最新技术水平,预训练的摘要模型,并发现SFT在CNN/DailyMail数据集上优于知识蒸馏和伪标记,但在更抽象的XSUM数据集上表现不佳。PyTorch代码和不同大小的检查点可以通过Hugging Face Transformers获得,网址为http://tiny.cc/4iy0tz。
Compression of Generative Pre-trained Language Models via Quantization
The University of Hong Kong(香港大学);Huawei Noah’s Ark Lab(华为诺亚方舟实验室)
生成式预训练语言模型(PLM)的规模不断增大,极大地增加了对模型压缩的需求。尽管有各种方法来压缩BERT或其变体,但很少有人尝试压缩生成PLM,并且潜在的困难仍然不清楚。在本文中,我们压缩生成PLM量化。我们发现,以前的量化方法失败的生成任务,由于\textit{homogeneous word embeddings}所造成的能力降低,和\textit{varied distribution of weights}。相应地,我们提出了一个令牌级的对比蒸馏学习可区分的词嵌入,和一个模块的动态缩放,使量化器适应不同的模块。各种任务的实证结果表明,我们提出的方法优于国家的最先进的压缩方法生成PLM的一个明确的保证金。与全精度模型相比,我们在GPT-2和BART上分别实现了14.4倍和13.4倍的压缩率。
OnlineQAT: On-Policy Distillation for Ultra-Low-Bit Large Language Models
The Hong Kong Polytechnic University(香港理工大学);Sun Yat-sen University(中山大学);PolyU-Daya Bay Technology and Innovation Research Institute(香港理工大学大亚湾技术创新研究院)