从神经网络中提取知识

Distilling a Neural Network Into a Soft Decision Tree

Deep neural networks have proved to be a very effective way to perform classification tasks. They excel when the input data is high dimensional, the relationship between the input and the output is complicated, and the number of labeled training examples is large. But it is hard to explain why a learned network makes a particular classification decision on a particular test case. This is due to their reliance on distributed hierarchical representations. If we could take the knowledge acquired by the neural net and express the same knowledge in a model that relies on hierarchical decisions inst…

Distilling neural networks into skipgram-level decision lists

Several previous studies on explanation for recurrent neural networks focus on approaches that find the most important input segments for a network as its explanations. In that case, the manner in which these input segments combine with each other to form an explanatory pattern remains unknown. To overcome this, some previous work tries to find patterns (called rules) in the data that explain neural outputs. However, their explanations are often insensitive to model parameters, which limits the scalability of text explanations. To overcome these limitations, we propose a pipeline to explain RN…

Graph-based Knowledge Distillation by Multi-head Attention Network

Knowledge distillation (KD) is a technique to derive optimal performance from a small student network (SN) by distilling knowledge of a large teacher network (TN) and transferring the distilled knowledge to the small SN. Since a role of convolutional neural network (CNN) in KD is to embed a dataset so as to perform a given task well, it is very important to acquire knowledge that considers intra-data relations. Conventional KD methods have concentrated on distilling knowledge in data units. To our knowledge, any KD methods for distilling information in dataset units have not yet been proposed.…

Distilling Knowledge from Ensembles of Acoustic Models for Joint CTC-Attention End-to-End Speech Recognition

Knowledge distillation has been widely used to compress existing deep learning models while preserving the performance on a wide range of applications. In the specific context of Automatic Speech Recognition (ASR), distillation from ensembles of acoustic models has recently shown promising results in increasing recognition performance. In this paper, we propose an extension of multi-teacher distillation methods to joint CTC-attention end-to-end ASR systems. We also introduce three novel distillation strategies. The core intuition behind them is to integrate the error rate metric to the teacher…

提取模型知识

Distilling Model Knowledge

性能最好的机器学习系统，如深度神经网络、大型集成和复杂的概率图模型，存储成本高、评估速度慢，而且难以集成到更大的系统中。理想情况下，我们希望用性能同样好的更简单的模型来取代这些繁琐的模型。在这篇论文中，我们研究了知识蒸馏，即提取包含在复杂模型中的知识并将其注入到更方便的模型中的思想。我们提出了一个知识蒸馏的一般框架，我们选择的一个方便的模型通过观察复杂模型的行为来学习如何模仿复杂模型，并且当它无法复制它时会受到惩罚。我们在三个不同的机器学习应用程序的背景下开发我们的框架：（a）模型压缩，其中我们将大型判别模型（例如神经网络的集合）压缩成小得多的模型;（b）用于贝叶斯推断的紧凑预测分布，其中我们将大量MCMC样本以封闭形式转换为紧凑预测分布;（c）难以处理的生成模型，其中我们将RBM等不可归一化的模型转换为NAD等易处理的模型。我们以新颖的技术和想法为艺术的状态做出贡献。在模型压缩中，我们描述并实现了导数匹配，这允许在数据稀缺时进行更好的蒸馏。在紧凑的预测分布中，我们引入了在线蒸馏，这可以显著节省内存。最后，在难处理的生成模型中，我们展示了如何使用蒸馏模型来鲁棒地估计原始模型的难处理的数量，例如其难处理的分区函数。

On Self-Distilling Graph Neural Network

Recently, the teacher-student knowledge distillation framework has demonstrated its potential in training Graph Neural Networks (GNNs). However, due to the difficulty of training over-parameterized GNN models, one may not easily obtain a satisfactory teacher model for distillation. Furthermore, the inefficient training process of teacher-student knowledge distillation also impedes its applications in GNN models. In this paper, we propose the first teacher-free knowledge distillation method for GNNs, termed GNN Self-Distillation (GNN-SD), that serves as a drop-in replacement of the standard tra…

Cracking the Black Box: Distilling Deep Sports Analytics

This paper addresses the trade-off between Accuracy and Transparency for deep learning applied to sports analytics. Neural nets achieve great predictive accuracy through deep learning, and are popular in sports analytics. But it is hard to interpret a neural net model and harder still to extract actionable insights from the knowledge implicit in it. Therefore, we built a simple and transparent model that mimics the output of the original deep learning model and represents the learned knowledge in an explicit interpretable way. Our mimic model is a linear model tree, which combines a collection…

Distilling BERT into Simple Neural Networks with Unlabeled Transfer Data

Recent advances in pre-training huge models on large amounts of text through self supervision have obtained state-of-the-art results in various natural language processing tasks. However, these huge and expensive models are difficult to use in practise for downstream tasks. Some recent efforts use knowledge distillation to compress these models. However, we see a gap between the performance of the smaller student models as compared to that of the large teacher. In this work, we leverage large amounts of in-domain unlabeled transfer data in addition to a limited amount of labeled training insta…

Distilling and Transferring Knowledge via cGAN-generated Samples for Image Classification and Regression

Knowledge distillation (KD) has been actively studied for image classification tasks in deep learning, aiming to improve the performance of a student based on the knowledge from a teacher. However, applying KD in image regression with a scalar response variable has been rarely studied, and there exists no KD method applicable to both classification and regression tasks yet. Moreover, existing KD methods often require a practitioner to carefully select or adjust the teacher and student architectures, making these methods less flexible in practice. To address the above problems in a unified way,…