Electroencephalogram (EEG) has been one of the common neuromonitoring modalities for real-world brain-computer interfaces (BCIs) because of its non-invasiveness, low cost, and high temporal resolution. Recently, light-weight and portable EEG wearable devices based on low-density montages have increased the convenience and usability of BCI applications. However, loss of EEG decoding performance is often inevitable due to reduced number of electrodes and coverage of scalp regions of a low-density EEG montage. To address this issue, we introduce knowledge distillation (KD), a learning mechanism developed for transferring knowledge/information between neural network models, to enhance the performance of low-density EEG decoding. Our framework includes a newly proposed similarity-keeping (SK) teacher-student KD scheme that encourages a low-density EEG student model to acquire the inter-sample similarity as in a pre-trained teacher model trained on high-density EEG data. The experimental results validate that our SK-KD framework consistently improves motor-imagery EEG decoding accuracy when number of electrodes deceases for the input EEG data. For both common low-density headphone-like and headband-like montages, our method outperforms state-of-the-art KD methods across various EEG decoding model architectures. As the first KD scheme developed for enhancing EEG decoding, we foresee the proposed SK-KD framework to facilitate the practicality of low-density EEG-based BCI in real-world applications.
University of Moratuwa（莫拉图瓦大学）；Harvard University（哈佛大学）；Aarhus University（奥胡斯大学）；Florida International University（佛罗里达国际大学）
Sleep plays a crucial role in the well-being of human lives. Traditional sleep studies using Polysomnography are associated with discomfort and often lower sleep quality caused by the acquisition setup. Previous works have focused on developing less obtrusive methods to conduct high-quality sleep studies, and ear-EEG is among popular alternatives. However, the performance of sleep staging based on ear-EEG is still inferior to scalp-EEG based sleep staging. In order to address the performance gap between scalp-EEG and ear-EEG based sleep staging, we propose a cross-modal knowledge distillation …
Multi-Branch Mutual-Distillation Transformer for EEG-Based Seizure Subtype Classification
Huazhong University of Science and Technology（华中科技大学）；Dongfeng Corporation Research & Development Institute（东风公司研发院）；China Electronic System Technology Co., Ltd.（中国电子系统技术有限公司）；Southern University of Science and Technology（南方科技大学）
An electroencephalogram (EEG) signal is currently accepted as a standard for automatic sleep staging. Lately, Near-human accuracy in automated sleep staging has been achievable by Deep Learning (DL) based approaches, enabling multi-fold progress in this area. However, An extensive and expensive clinical setup is required for EEG based sleep staging. Additionally, the EEG setup being obtrusive in nature and requiring an expert for setup adds to the inconvenience of the subject under study, making it adverse in the point of care setting. An unobtrusive and more suitable alternative to EEG is Ele…
DistilCLIP-EEG: Enhancing Epileptic Seizure Detection Through Multi-modal Learning and Knowledge Distillation
Aliyun School of Big Data, Changzhou University（阿里云大数据学院，长洲大学）；Department of Computing, Xi’an JiaoTong-Liverpool University（计算系，西安交通大学-利物浦大学）；Department of Computer Science, University of Liverpool（计算机科学系，利物浦大学）；Center for Artificial Intelligence in Agriculture, Fujian Agriculture and Forestry University（农业人工智能中心，福建农林大学）
Pre-Training Graph Contrastive Masked Autoencoders are Strong Distillers for EEG
Department of Electrical and Computer Engineering, Lehigh University, Bethlehem, PA, USA（电气与计算机工程系，莱维大学）；Department of Bioengineering, Lehigh University, Bethlehem, PA, USA（生物工程系，莱维大学）；Center for Neuroscience Research, Children’s National Hospital, Washington, DC, USA（神经科学研究中心，儿童国家医院）；Department of Computer Science and Engineering, Lehigh University, Bethlehem, PA, USA（计算机科学与工程系，莱维大学）
DLink: Distilling Layer-wise and Dominant Knowledge from EEG Foundation Models
Xiamen University（厦门大学）；Nanyang Technological University（南洋理工大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Southeast University（东南大学）
Canine EEG Helps Human: Cross-Species and Cross-Modality Epileptic Seizure Detection via Multi-Space Alignment
Ministry of Education Key Laboratory of Image Processing and Intelligent Control, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（教育部图像处理与智能控制重点实验室，人工智能与自动化学院，华中科技大学）；Hubei Key Laboratory of Brain-inspired Intelligent Systems, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（湖北省脑启发智能系统重点实验室，人工智能与自动化学院，华中科技大学）
Affective computing with Electroencephalogram (EEG) is a challenging task that requires cumbersome models to effectively learn the information contained in large-scale EEG signals, causing difficulties for real-time smart-device deployment. In this paper, we propose a novel knowledge distillation pipeline to distill EEG representations via capsule-based architectures for both classification and regression tasks. Our goal is to distill information from a heavy model to a lightweight model for subject-specific tasks. To this end, we first pre-train a large model (teacher network) on large number…
Foundation Model Guided Dual-Branch Co-Adaptation for Source-Free EEG Decoding
College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；College of Artifical Intelligence and Automation, Hohai University（河海大学人工智能与自动化学院）；College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics（南京航空航天大学人工智能学院）；Brainnetcome Center, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所脑网络中心）；Information Systems Technology and Design, Singapore University of Technology and Design（新加坡科技设计大学信息系统技术与设计）
It is challenging to perform lifelong language learning (LLL) on a stream of different tasks without any performance degradation comparing to the multi-task counterparts. To address this issue, we present Lifelong Language Knowledge Distillation (L2KD), a simple but efficient method that can be easily applied to existing LLL architectures in order to mitigate the degradation. Specifically, when the LLL model is trained on a new task, we assign a teacher model to first learn the new task, and pass the knowledge to the LLL model via knowledge distillation. Therefore, the LLL model can better ada…
NTT Media Intelligence Laboratories, NTT Corporation（NTT公司NTT媒体情报实验室）
This paper presents a novel knowledge distillation method for dialogue sequence labeling. Dialogue sequence labeling is a supervised learning task that estimates labels for each utterance in the target dialogue document, and is useful for many applications such as dialogue act estimation. Accurate labeling is often realized by a hierarchically-structured large model consisting of utterance-level and dialogue-level networks that capture the contexts within an utterance and between utterances, respectively. However, due to its large model size, such a model cannot be deployed on resource-constra…
Current deep learning models often suffer from catastrophic forgetting of old knowledge when continually learning new knowledge. Existing strategies to alleviate this issue often fix the trade-off between keeping old knowledge (stability) and learning new knowledge (plasticity). However, the stability-plasticity trade-off during continual learning may need to be dynamically changed for better model performance. In this paper, we propose two novel ways to adaptively balance model stability and plasticity. The first one is to adaptively integrate multiple levels of old knowledge and transfer it …
LIB-KD: Teaching Inductive Bias for Efficient Vision Transformer Distillation and Compression
Department of Electrical Engineering Indian Institute of Technology Delhi（电子工程系印度理工学院德里）；Yogananda School of AI, Computers and Data Sciences（Yoganananda人工智能、计算机与数据科学学院）
Ping An Technology (Shenzhen) Co., Ltd.（平安科技（深圳）有限公司）；University of Science and Technology of China（中国科学技术大学）；Shanghai International Studies University（上海外国语大学）；Peking University（北京大学）
Deep neural networks can capture the intricate interaction history information between queries and documents, because of their many complicated nonlinear units, allowing them to provide correct search recommendations. However, service providers frequently face more complex obstacles in real-world circumstances, such as deployment cost constraints and fairness requirements. Knowledge distillation, which transfers the knowledge of a well-trained complex model (teacher) to a simple model (student), has been proposed to alleviate the former concern, but the best current distillation methods focus …
University of Alabama in Huntsville（阿拉巴马大学亨茨维尔分校）；Columbia University（哥伦比亚大学）
Detecting anomalies has become increasingly critical to the financial service industry. Anomalous events are often indicative of illegal activities such as fraud, identity theft, network intrusion, account takeover, and money laundering. Financial anomaly detection use cases face serious challenges due to the dynamic nature of the underlying patterns especially in adversarial environments such as constantly changing fraud tactics. While retraining the models with the new patterns is absolutely essential; keeping up with the rapid changes introduces other challenges as it moves the model away f…
RPTU Kaiserslautern-Landau（莱茵兰-普法尔茨州凯泽斯劳滕-兰道应用技术大学）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Heidelberg University（海德堡大学）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心（DFKI））
Morphological atlases are an important tool in organismal studies, and modern high-throughput Computed Tomography (CT) facilities can produce hundreds of full-body high-resolution volumetric images of organisms. However, creating an atlas from these volumes requires accurate organ segmentation. In the last decade, machine learning approaches have achieved incredible results in image segmentation tasks, but they require large amounts of annotated data for training. In this paper, we propose a self-training framework for multi-organ segmentation in tomographic images of Medaka fish. We utilize t…
In recent years, deeper and wider neural networks have shown excellent performance in computer vision tasks, while their enormous amount of parameters results in increased computational cost and overfitting. Several methods have been proposed to compress the size of the networks without reducing network performance. Network pruning can reduce redundant and unnecessary parameters from a network. Knowledge distillation can transfer the knowledge of deeper and wider networks to smaller networks. The performance of the smaller network obtained by these methods is bounded by the predefined network.…
CMCRD: Cross-Modal Contrastive Representation Distillation for Emotion Recognition
Key Laboratory of Image Processing and Intelligent Control, Ministry of Education, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（图像处理与智能控制重点实验室、教育部、自动化学院、华中科技大学）；Wuhan Institute of Digital Engineering（武汉数字工程研究所）；Shanghai Jiao Tong University（上海交通大学）
Nanjing University（南京大学）；Nanjing University of Posts and Telecommunications（南京邮电大学）
Although deep reinforcement learning has become a universal solution for complex control tasks, its real-world applicability is still limited because lacking security guarantees for policies. To address this problem, we propose Boundary Characterization via the Minimum Experience Retention (BCMER), an end-to-end Interpretable Policy Distillation (IPD) framework. Unlike previous IPD approaches, BCMER distinguishes the importance of experiences and keeps a minimal but critical experience pool with almost no loss of policy similarity. Specifically, the proposed BCMER contains two basic steps. Fir…
FedL2T: Personalized Federated Learning with Two-Teacher Distillation for Seizure Prediction
Key Laboratory of Smart Manufacturing in Energy Chemical Process, Ministry of Education East China University of Science（能源化学过程智能制造重点实验室，教育部东华大学）；School of Information Science（信息科学学院）；Engineering East China University of Science（工程东华大学）
Deep neural networks (DNN) have recently been widely used in speaker recognition systems, achieving state-of-the-art performance on various benchmarks. The x-vector architecture is especially popular in this research community, due to its excellent performance and manageable computational complexity. In this paper, we present the lrx-vector system, which is the low-rank factorized version of the x-vector embedding network. The primary objective of this topology is to further reduce the memory requirement of the speaker recognition system. We discuss the deployment of knowledge distillation for…
Privacy-preserving machine learning has become a key conundrum for multi-party artificial intelligence. Federated learning (FL) and Split Learning (SL) are two frameworks that enable collaborative learning while keeping the data private (on device). In FL, each data holder trains a model locally and releases it to a central server for aggregation. In SL, the clients must release individual cut-layer activations (smashed data) to the server and wait for its response (during both inference and back propagation). While relevant in several settings, both of these schemes have a high communication …
Lomonosov Moscow State University（莫斯科国立大学（罗蒙诺索夫莫斯科国立大学））
Data distillation is the problem of reducing the volume oftraining data while keeping only the necessary information. With thispaper, we deeper explore the new data distillation algorithm, previouslydesigned for image data. Our experiments with tabular data show thatthe model trained on distilled samples can outperform the model trainedon the original dataset. One of the problems of the considered algorithmis that produced data has poor generalization on models with differenthyperparameters. We show that using multiple architectures during distillation can help overcome this problem.
Recent advances in deep learning have facilitated the demand of neural models for real applications. In practice, these applications often need to be deployed with limited resources while keeping high accuracy. This paper touches the core of neural models in NLP, word embeddings, and presents a new embedding distillation framework that remarkably reduces the dimension of word embeddings without compromising accuracy. A novel distillation ensemble approach is also proposed that trains a high-efficient student model using multiple teacher models. In our approach, the teacher models play roles on…
Feature Distillation is the Better Choice for Model-Heterogeneous Federated Learning
School of Computer Science and Technology, Huazhong University of Science and Technology（华中科技大学计算机科学与技术学院）；Mohamed bin Zayed University of Artificial Intelligence（莫扎德·本·扎耶德人工智能大学）；International School, Beijing University of Posts and Telecommunications（北京邮电大学国际学院）；Department of Computing, The Hong Kong Polytechnic University（香港理工大学计算机系）
Towards Collaborative Fairness in Federated Learning Under Imbalanced Covariate Shift
The Pennsylvania State University（宾夕法尼亚州立大学）；State University of New York at Albany（纽约州立大学阿尔巴尼分校）；University of Minnesota Twin Cities（明尼苏达大学双城分校）；Dalian University of Technology（大连理工大学）
LBLLM: Lightweight Binarization of Large Language Models via Three-Stage Distillation
MAIS, Institute of Automation, Chinese Academy of Sciences（自动化研究所，中国科学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Central Media Technology Institute, Huawei（华为中央媒体技术研究所）
SAFE: Secure and Accurate Federated Learning for Privacy-Preserving Brain-Computer Interfaces
Key Laboratory of the Ministry of Education for Image Processing and Intelligent Control, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（教育部图像处理与智能控制重点实验室，人工智能与自动化学院，华中科技大学）；Zhongguancun Academy（中关村学院）
