TextBrewer: An Open-Source Knowledge Distillation Toolkit for Natural Language Processing

EasyNLP: A Comprehensive and Easy-to-use Toolkit for Natural Language Processing

The success of Pre-Trained Models (PTMs) has reshaped the development of Natural Language Processing (NLP). Yet, it is not easy to obtain high-performing models and deploy them online for industrial practitioners. To bridge this gap, EasyNLP is designed to make it easy to build NLP applications, which supports a comprehensive suite of NLP algorithms. It further features knowledge-enhanced pre-training, knowledge distillation and few-shot learning functionalities for large-scale PTMs, and provides a unified framework of model training, inference and deployment for real-world applications. Curre…

The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understanding

We present MT-DNN, an open-source natural language understanding (NLU) toolkit that makes it easy for researchers and developers to train customized deep learning models. Built upon PyTorch and Transformers, MT-DNN is designed to facilitate rapid customization for a broad spectrum of NLU tasks, using a variety of objectives (classification, regression, structured prediction) and text encoders (e.g., RNNs, BERT, RoBERTa, UniLM). A unique feature of MT-DNN is its built-in support for robust and transferable learning using the adversarial multi-task learning paradigm. To enable efficient producti…

PHI-S：面向无标签多教师蒸馏的分布平衡

PHI-S: Distribution Balancing for Label-Free Multi-Teacher Distillation

针对无标签异构多教师知识蒸馏中的教师激活分布失配问题，本文提出基于 Hadamard 矩阵的各向同性 PHI-S 标准化方法，并在多种方法中取得最佳学生模型。

EasyDistill：用于大型语言模型有效知识蒸馏的综合工具包

EasyDistill: A Comprehensive Toolkit for Effective Knowledge Distillation of Large Language Models

本文提出EasyDistill工具包，支持大型语言模型黑白盒知识蒸馏，集成数据合成、微调、排序优化与强化学习等功能，并提供蒸馏模型、工业方案及开源数据集，无缝接入阿里云PAI平台。

N-LTP: An Open-source Neural Language Technology Platform for Chinese

We introduce \texttt{N-LTP}, an open-source neural language technology platform supporting six fundamental Chinese NLP tasks: {lexical analysis} (Chinese word segmentation, part-of-speech tagging, and named entity recognition), {syntactic parsing} (dependency parsing), and {semantic parsing} (semantic dependency parsing and semantic role labeling). Unlike the existing state-of-the-art toolkits, such as \texttt{Stanza}, that adopt an independent model for each task, \texttt{N-LTP} adopts the multi-task framework by using a shared pre-trained model, which has the advantage of capturing the share…