Knowledge transfer among multiple networks using their outputs or intermediate activations have evolved through extensive manual design from a simple teacher-student approach (knowledge distillation) to a bidirectional cohort one (deep mutual learning). The key factors of such knowledge transfer involve the network size, the number of networks, the transfer direction, and the design of the loss function. However, because these factors are enormous when combined and become intricately entangled, the methods of conventional knowledge transfer have explored only limited combinations. In this paper, we propose a new graph-based approach for more flexible and diverse combinations of knowledge transfer. To achieve the knowledge transfer, we propose a novel graph representation called knowledge transfer graph that provides a unified view of the knowledge transfer and has the potential to represent diverse knowledge transfer patterns. We also propose four gate functions that are introduced into loss functions. The four gates, which control the gradient, can deliver diverse combinations of knowledge transfer. Searching the graph structure enables us to discover more effective knowledge transfer methods than a manually designed one. Experimental results on the CIFAR-10, -100, and Tiny-ImageNet datasets show that the proposed method achieved significant performance improvements and was able to find remarkable graph structures.
Enhancing Scene Classification in Cloudy Image Scenarios: A Collaborative Transfer Method with Information Regulation Mechanism using Optical Cloud-Covered and SAR Remote Sensing Images
School of Geosciences and Info-Physics, Central South University（中南大学地球科学与信息物理学院）；Faculty of Geo-Information Science and Earth Observation (ITC), University of Twente（特文特大学地球信息科学与地球观测学院）
Face Completion with Semantic Knowledge and Collaborative Adversarial Learning
University of Rochester（罗切斯特大学）；Siemens Healthineers（西门子医疗）；Tencent X-Lab（腾讯X实验室）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）
SCORE: SubDistribution-aware Collaborative Knowledge Reinforcing for Cloth-Hybrid Lifelong Person Re-Identification
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Intelligent Science & Technology Academy of CASIC（中国航天科工集团智能科技研究院）
Unsupervised Domain Adaptation for Cross-Modality Retinal Vessel Segmentation via Disentangling Representation Style Transfer and Collaborative Consistency Learning
Southern University of Science and Technology（南方科技大学）；The University of Hong Kong（香港大学）
Seeking Common Ground While Reserving Differences: Multiple Anatomy Collaborative Framework for Undersampled MRI Reconstruction
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Tencent AI Lab（腾讯人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）；University of Texas at Arlington（德克萨斯大学阿灵顿分校）
CNN-Transformer Rectified Collaborative Learning for Medical Image Segmentation
School of Information and Communication Engineering, Dalian University of Technology（大连理工大学信息与通信工程学院）；Key Laboratory for Ubiquitous Network and Service Software of Liaoning Province（辽宁省泛在网络与服务软件重点实验室）；DUT-RU International School of Information Science and Engineering, Dalian University of Technology（大连理工大学-鲁东大学国际信息科学与工程学院）；Affiliated Zhongshan Hospital of Dalian University（大连大学附属中山医院）
