Knowledge distillation (KD) has proved to be an effective approach for deep neural network compression, which learns a compact network (student) by transferring the knowledge from a pre-trained, over-parameterized network (teacher). In traditional KD, the transferred knowledge is usually obtained by feeding training samples to the teacher network to obtain the class probabilities. However, the original training dataset is not always available due to storage costs or privacy issues. In this study, we propose a novel data-free KD approach by modeling the intermediate feature space of the teacher with a multivariate normal distribution and leveraging the soft targeted labels generated by the distribution to synthesize pseudo samples as the transfer set. Several student networks trained with these synthesized transfer sets present competitive performance compared to the networks trained with the original training set and other data-free KD approaches.
Zhejiang University（浙江大学）；National University of Singapore（新加坡国立大学）；Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies（阿里巴巴-浙江大学前沿技术联合研究中心）
Data-free knowledge distillation (DFKD) has recently been attracting increasing attention from research communities, attributed to its capability to compress a model only using synthetic data. Despite the encouraging results achieved, state-of-the-art DFKD methods still suffer from the inefficiency of data synthesis, making the data-free training process extremely time-consuming and thus inapplicable for large-scale tasks. In this work, we introduce an efficacious scheme, termed as FastDFKD, that allows us to accelerate DFKD by a factor of orders of magnitude. At the heart of our approach is a…
Prior work has shown that, on small amounts of training data, syntactic neural language models learn structurally sensitive generalisations more successfully than sequential language models. However, their computational complexity renders scaling difficult, and it remains an open question whether structural biases are still necessary when sequential models have access to ever larger amounts of training data. To answer this question, we introduce an efficient knowledge distillation (KD) technique that transfers knowledge from a syntactic language model trained on a small corpus to an LSTM langu…
Cambridge, United Kingdom（英国剑桥）；Cape Town, South Africa（南非开普敦）
This paper demonstrates the application of reinforcement learning (RL) to process synthesis by presenting Distillation Gym, a set of RL environments in which an RL agent is tasked with designing a distillation train, given a user defined multi-component feed stream. Distillation Gym interfaces with a process simulator (COCO and ChemSep) to simulate the environment. A demonstration of two distillation problem examples are discussed in this paper (a Benzene, Toluene, P-xylene separation problem and a hydrocarbon separation problem), in which a deep RL agent is successfully able to learn within D…
Despite their accuracy, neural network-based classifiers are still prone to manipulation through adversarial perturbations. Those perturbations are designed to be misclassified by the neural network, while being perceptually identical to some valid input. The vast majority of attack methods rely on white-box conditions, where the attacker has full knowledge of the attacked network's parameters. This allows the attacker to calculate the network's loss gradient with respect to some valid input and use this gradient in order to create an adversarial example. The task of blocking white-box attacks…
Deep Neural Networks (DNNs) are vulnerable to adversarial attacks, especially white-box targeted attacks. One scheme of learning attacks is to design a proper adversarial objective function that leads to the imperceptible perturbation for any test image (e.g., the Carlini-Wagner (C&W) method). Most methods address targeted attacks in the Top-1 manner. In this paper, we propose to learn ordered Top-k attacks (k>= 1) for image classification tasks, that is to enforce the Top-k predicted labels of an adversarial example to be the k (randomly) selected and ordered labels (the ground-truth label is…
Apple（苹果公司）；University of California, San Diego（加州大学圣地亚哥分校）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；University of Pennsylvania（宾夕法尼亚大学）
Novel view synthesis from a single image requires inferring occluded regions of objects and scenes whilst simultaneously maintaining semantic and physical consistency with the input. Existing approaches condition neural radiance fields (NeRF) on local image features, projecting points to the input image plane, and aggregating 2D features to perform volume rendering. However, under severe occlusion, this projection fails to resolve uncertainty, resulting in blurry renderings that lack details. In this work, we propose NerfDiff, which addresses this issue by distilling the knowledge of a 3D-awar…
Diversity-Driven Synthesis: Enhancing Dataset Distillation through Directed Weight Adjustment
Centre for Frontier AI Research (CFAR)（前沿人工智能研究中心）；Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局）；Institute of High Performance Computing（高性能计算研究所）；XiDian University（西安电子科技大学）；National University of Singapore（新加坡国立大学）；Hubei University（湖北大学）
This paper shows the implementation of reinforcement learning (RL) in commercial flowsheet simulator software (Aspen Plus V12) for designing and optimising a distillation sequence. The aim of the SAC agent was to separate a hydrocarbon mixture in its individual components by utilising distillation. While doing so it tries to maximise the profit produced by the distillation sequence. All actions of the agent were set by the SAC agent in Python and communicated in Aspen Plus via an API. Here the distillation column was simulated by use of the build-in RADFRAC column. With this a connection was e…
LiBaGS: Lightweight Boundary Gap Synthesis for Targeted Synthetic Data Selection
Department of Computer Science（计算机科学系）；University of Toronto（多伦多大学）；The Hospital for Sick Children UHN（Sick Children医院 UHN）；KITE Research Institute（KITE研究机构）；T-CAIREM Vector Institute（T-CAIREM向量研究所）；Vector Institute（向量研究所）
Columbia University（哥伦比亚大学）；Microsoft Research（微软研究院）
Contrastive language-image pretraining (CLIP) links vision and language modalities into a unified embedding space, yielding the tremendous potential for vision-language (VL) tasks. While early concurrent works have begun to study this potential on a subset of tasks, important questions remain: 1) What is the benefit of CLIP on unstudied VL tasks? 2) Does CLIP provide benefit in low-shot or domain-shifted scenarios? 3) Can CLIP improve existing approaches without impacting inference or pretraining complexity? In this work, we seek to answer these questions through two key contributions. First, …
Find Your Optimal Teacher: Personalized Data Synthesis via Router-Guided Multi-Teacher Distillation
The University of Hong Kong（香港大学）；Simon Fraser University（西蒙菲莎大学）；University of California, Los Angeles（加州大学洛杉矶分校）；Dartmouth College（达特茅斯学院）；University of Maryland, Baltimore County（马里兰大学巴尔的摩县分校）；Tencent（腾讯）
Conformal Disentanglement and Latent-Space Curation: A Neural Framework for Perspective Synthesis, Differentiation and Targeted Generation
Department of Applied Mathematics and Statistics, Johns Hopkins University（应用数学与统计学系，约翰霍普金斯大学）；Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）；Faculty of Science, Technology and Medicine, University of Luxembourg（科学、技术与医学学院，卢森堡大学）；Department of Civil and Systems Engineering, Johns Hopkins University（土木与系统工程系，约翰霍普金斯大学）；Department of Chemical and Biomolecular Engineering, Johns Hopkins University（化学与生物分子工程系，约翰霍普金斯大学）
Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所基础模型研究中心）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；National University of Singapore（新加坡国立大学）；Tencent（腾讯）；Wuhan AI Research（武汉人工智能研究院）
STEPS: Selective On-Policy Self-Distillation for Reasoning
State Key Laboratory of Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）；School of Intelligence Science and Technology, Nanjing University（南京大学智能科学与技术学院）；AMAP, Alibaba Group（阿里集团AMAP）；University of Wisconsin–Madison（威斯康星大学麦迪逊分校）；Tsinghua University（清华大学）
MDM: Advancing Multi-Domain Distribution Matching for Automatic Modulation Recognition Dataset Synthesis
College of Information Engineering, Zhejiang University of Technology（浙江工业大学信息工程学院）；Institute of Cyberspace Security, Zhejiang University of Technology（浙江工业大学网络空间安全研究院）；Institute of China Electronics Technology Group Corporation 36th Research Institute（中国电子科技集团公司第三十六研究所）；College of Information And Communication Engineering, Harbin Engineering University（哈尔滨工程大学信息与通信工程学院）；National Key Laboratory of Electromagnetic Space Security（电磁空间安全全国重点实验室）
Temporal Saliency-Guided Distillation: A Scalable Framework for Distilling Video Datasets
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Peng Cheng Laboratory（鹏城实验室）；Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；Shanghai Jiao Tong University（上海交通大学）
Technical University of Munich（慕尼黑工业大学）；Weihenstephan-Triesdorf University of Applied Sciences（魏恩施蒂芬-特里斯多夫应用科学大学）
Automated flowsheet synthesis is an important field in computer-aided process engineering. The present work demonstrates how reinforcement learning can be used for automated flowsheet synthesis without any heuristics of prior knowledge of conceptual design. The environment consists of a steady-state flowsheet simulator that contains all physical knowledge. An agent is trained to take discrete actions and sequentially built up flowsheets that solve a given process problem. A novel method named SynGameZero is developed to ensure good exploration schemes in the complex problem. Therein, flowsheet…
