Transfer Heterogeneous Knowledge Among Peer-to-Peer Teammates: A Model Distillation Approach
Huazhong University of Science and Technology（华中科技大学）；Zhejiang University（浙江大学）；Peking University（北京大学）；University of Arkansas, Fayetteville（阿肯色大学费耶特维尔分校）
Zeyue Xue, Shuang Luo, Chao Wu, Pan Zhou, Kaigui Bian, Wei Du
Peer-to-peer knowledge transfer in distributed environments has emerged as a promising method since it could accelerate learning and improve team-wide performance without relying on pre-trained teachers in deep reinforcement learning. However, for traditional peer-to-peer methods such as action advising, they have encountered difficulties in how to efficiently expressed knowledge and advice. As a result, we propose a brand new solution to reuse experiences and transfer value functions among multiple students via model distillation. But it is still challenging to transfer Q-function directly since it is unstable and not bounded. To address this issue confronted with existing works, we adopt Categorical Deep Q-Network. We also describe how to design an efficient communication protocol to exploit heterogeneous knowledge among multiple distributed agents. Our proposed framework, namely Learning and Teaching Categorical Reinforcement (LTCR), shows promising performance on stabilizing and accelerating learning progress with improved team-wide reward in four typical experimental environments.
Zhejiang University（浙江大学）；ZJU-LianlianPay Joint Research Center（浙江大学-连连支付联合研究中心）；Zhejiang University of Technology（浙江工业大学）
Distillation is an effective knowledge-transfer technique that uses predicted distributions of a powerful teacher model as soft targets to train a less-parameterized student model. A pre-trained high capacity teacher, however, is not always available. Recently proposed online variants use the aggregated intermediate predictions of multiple student models as targets to train each student model. Although group-derived targets give a good recipe for teacher-free distillation, group members are homogenized quickly with simple aggregation functions, leading to early saturated solutions. In this wor…
Policy distillation, which transfers a teacher policy to a student policy has achieved great success in challenging tasks of deep reinforcement learning. This teacher-student framework requires a well-trained teacher model which is computationally expensive. Moreover, the performance of the student model could be limited by the teacher model if the teacher model is not optimal. In the light of collaborative learning, we study the feasibility of involving joint intellectual efforts from diverse perspectives of student models. In this work, we introduce dual policy distillation(DPD), a student-s…
Convergence Theory of Knowledge Distillation in Asynchronous P2P Gossip Learning Network
University of California, Santa Cruz（加州大学圣克鲁兹分校）；China University of Petroleum（中国石油大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；City University of Hong Kong（香港城市大学）；University of Virginia（弗吉尼亚大学）
FedSKD: Aggregation-free Model-heterogeneous Federated Learning via Multi-dimensional Similarity Knowledge Distillation for Medical Image Classification
1 Integrated Graduate School of Medicine, Engineering；Agricultural Sciences, University of Yamanashi, Japan
This paper describes that semi-supervised learning called peer collaborative learning (PCL) can be applied to the polyphonic sound event detection (PSED) task, which is one of the tasks in the Detection and Classification of Acoustic Scenes and Events (DCASE) challenge. Many deep learning models have been studied to find out what kind of sound events occur where and for how long in a given audio clip. The characteristic of PCL used in this paper is the combination of ensemble-based knowledge distillation into sub-networks and student-teacher model-based knowledge distillation, which can train …
Beijing Institute of Technology（北京理工大学）；Peking University（北京大学）
Neural dialogue models suffer from low-quality responses when interacted in practice, demonstrating difficulty in generalization beyond training data. Recently, knowledge distillation has been used to successfully regularize the student by transferring knowledge from the teacher. However, the teacher and the student are trained on the same dataset and tend to learn similar feature representations, whereas the most general knowledge should be found through differences. The finding of general knowledge is further hindered by the unidirectional distillation, as the student should obey the teacher…
Benchmarking the Limits of In-Context Reinforcement Learning for Ad-Hoc Teamwork
C$^{2}$DL, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所C²DL实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；School of Future Technology, University of Chinese Academy of Sciences（中国科学院大学未来技术学院）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究所）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；University of Science and Technology of China（中国科学技术大学）；Qwen Team, Alibaba Group（阿里集团Qwen团队）
Tsinghua University（清华大学）；Fraunhofer Heinrich Hertz Institute（弗劳恩霍夫海因里希·赫兹研究所）
The recent advent of various forms of Federated Knowledge Distillation (FD) paves the way for a new generation of robust and communication-efficient Federated Learning (FL), where mere soft-labels are aggregated, rather than whole gradients of Deep Neural Networks (DNN) as done in previous FL schemes. This security-per-design approach in combination with increasingly performant Internet of Things (IoT) and mobile devices opens up a new realm of possibilities to utilize private data from industries as well as from individuals as input for artificial intelligence model training. Yet in previous …
Uncertainty-Aware Dual-Student Knowledge Distillation for Efficient Image Classification
Department of Electrical Engineering Indian Institute of Technology Bombay 21D070002（电子工程系 印度理工学院班加罗尔 21D070002）；Department of Electrical Engineering Indian Institute of Technology Bombay 210010010（电子工程系 印度理工学院班加罗尔 210010010）；Department of Electrical Engineering Indian Institute of Technology Bombay 210020023（电子工程系 印度理工学院班加罗尔 210020023）
Siemens AG, Corporate Technology（西门子公司企业技术部）；University of Munich（慕尼黑大学）
In recent years, many explanation methods have been proposed to explain individual classifications of deep neural networks. However, how to leverage the created explanations to improve the learning process has been less explored. As the privileged information, the explanations of a model can be used to guide the learning process of the model itself. In the community, another intensively investigated privileged information used to guide the training of a model is the knowledge from a powerful teacher model. The goal of this work is to leverage the self-explanation to improve the learning proces…
MIT-LIDS（麻省理工学院信息与决策系统实验室）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；IBM Research（IBM研究院）
Collective learning can be greatly enhanced when agents effectively exchange knowledge with their peers. In particular, recent work studying agents that learn to teach other teammates has demonstrated that action advising accelerates team-wide learning. However, the prior work has simplified the learning of advising policies by using simple function approximations and only considered advising with primitive (low-level) actions, limiting the scalability of learning and teaching to complex domains. This paper introduces a novel learning-to-teach framework, called hierarchical multiagent teaching…
UniOPSD: Unifying Outcome and Hindsight Feedback for Agentic Reinforcement Learning
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Peking University（北京大学）；Mininglamp Technology（明略科技）；Key Laboratory of Computing Power Network and Information Security, Ministry of Education（计算力网络与信息安全教育部重点实验室；山东省计算中心（齐鲁工业大学（山东省科学院）））；Shandong Computer Science Center, Qilu University of Technology (Shandong Academy of Sciences)（计算力互联网与服务计算重点实验室，山东省计算机科学基础研究中心）；Key Laboratory of Computing Power Internet and Service Computing, Shandong Fundamental Research Center for Computer Science
Advanced Research Lab NavInfo Europe（四维图新欧洲高级研究实验室）
Self-supervised learning solves pretext prediction tasks that do not require annotations to learn feature representations. For vision tasks, pretext tasks such as predicting rotation, solving jigsaw are solely created from the input data. Yet, predicting this known information helps in learning representations useful for downstream tasks. However, recent works have shown that wider and deeper models benefit more from self-supervised learning than smaller models. To address the issue of self-supervised pre-training of smaller models, we propose Distill-on-the-Go (DoGo), a self-supervised learni…
Graph neural networks (GNNs) have been widely used in many graph-based tasks such as node classification, link prediction, and node clustering. However, GNNs gain their performance benefits mainly from performing the feature propagation and smoothing across the edges of the graph, thus requiring sufficient connectivity and label information for effective propagation. Unfortunately, many real-world networks are sparse in terms of both edges and labels, leading to sub-optimal performance of GNNs. Recent interest in this sparse problem has focused on the self-training approach, which expands supe…
