In visual robot self-localization, graph-based scene representation and matching have recently attracted research interest as robust and discriminative methods for selflocalization. Although effective, their computational and storage costs do not scale well to large-size environments. To alleviate this problem, we formulate self-localization as a graph classification problem and attempt to use the graph convolutional neural network (GCN) as a graph classification engine. A straightforward approach is to use visual feature descriptors that are employed by state-of-the-art self-localization systems, directly as graph node features. However, their superior performance in the original self-localization system may not necessarily be replicated in GCN-based self-localization. To address this issue, we introduce a novel teacher-to-student knowledge-transfer scheme based on rank matching, in which the reciprocal-rank vector output by an off-the-shelf state-of-the-art teacher self-localization model is used as the dark knowledge to transfer. Experiments indicate that the proposed graph-convolutional self-localization network can significantly outperform state-of-the-art self-localization systems, as well as the teacher classifier.
Beijing University of Posts and Telecommunications（北京邮电大学）
In this paper, we aim to improve the performance of in-the-wild Facial Expression Recognition (FER) by exploiting semi-supervised learning. Large-scale labeled data and deep learning methods have greatly improved the performance of image recognition. However, the performance of FER is still not ideal due to the lack of training data and incorrect annotations (e.g., label noises). Among existing in-the-wild FER datasets, reliable ones contain insufficient data to train robust deep models while large-scale ones are annotated in lower quality. To address this problem, we propose a semi-supervised…
Boosting Light-Weight Depth Estimation Via Knowledge Distillation
Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳市人工智能与机器人研究院）；South China Normal University（华南师范大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Zhejiang University（浙江大学）；Tohoku University（东北大学（日本））
C2E: Boosting Ego-Only 3D Object Detection via Multi-Teacher Contrastive Knowledge Distillation
Fujian Key Laboratory of Urban Intelligent Sensing and Computing, Xiamen University（福建省城市智能感知与计算重点实验室，厦门大学）；Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（多媒体可信感知与高效计算教育部重点实验室，厦门大学）；Zhongguancun Academy（中关村学院）
School of Computing Sciences（计算科学学院）；University of East Anglia（东英吉利大学）
We propose SUB-Depth, a universal multi-task training framework for self-supervised monocular depth estimation (SDE). Depth models trained with SUB-Depth outperform the same models trained in a standard single-task SDE framework. By introducing an additional self-distillation task into a standard SDE training framework, SUB-Depth trains a depth network, not only to predict the depth map for an image reconstruction task, but also to distill knowledge from a trained teacher network with unlabelled data. To take advantage of this multi-task setting, we propose homoscedastic uncertainty formulatio…
University of Alabama at Birmingham Birmingham, United States；Johns Hopkins University Baltimore, United States；Dalian University of Technology Dalian, China；Aerospace Information Research Institute, CAS \& University of Chinese Academy of Sciences
While self-supervised representation learning (SSL) has proved to be effective in the large model, there is still a huge gap between the SSL and supervised method in the lightweight model when following the same solution. We delve into this problem and find that the lightweight model is prone to collapse in semantic space when simply performing instance-wise contrast. To address this issue, we propose a relation-wise contrastive paradigm with Relation Knowledge Distillation (ReKD). We introduce a heterogeneous teacher to explicitly mine the semantic information and transferring a novel relatio…
MapDistill: Boosting Efficient Camera-based HD Map Construction via Camera-LiDAR Fusion Model Distillation
Samsung R&D Institute China–Beijing（三星中国北京研发中心）；Beihang University（北京航空航天大学）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；The University of Sydney（悉尼大学）
3D object detection task from lidar or camera sensors is essential for autonomous driving. Pioneer attempts at multi-modality fusion complement the sparse lidar point clouds with rich semantic texture information from images at the cost of extra network designs and overhead. In this work, we propose a novel semantic passing framework, named SPNet, to boost the performance of existing lidar-based 3D detection models with the guidance of rich context painting, with no extra computation cost during inference. Our key design is to first exploit the potential instructive semantic knowledge within t…
Hefei University of Technology（合肥工业大学）；The University of Queensland（昆士兰大学）
Online Knowledge Distillation (OKD) improves the involved models by reciprocally exploiting the difference between teacher and student. Several crucial bottlenecks over the gap between them -- e.g., Why and when does a large gap harm the performance, especially for student? How to quantify the gap between teacher and student? -- have received limited formal study. In this paper, we propose Switchable Online Knowledge Distillation (SwitOKD), to answer these questions. Instead of focusing on the accuracy gap at test phase by the existing arts, the core idea of SwitOKD is to adaptively calibrate …
Krasovskii Institute of Mathematics and Mechanics（克拉斯诺夫斯基数学与力学研究所）；Ural Federal University（乌拉尔联邦大学）
We propose a response-based method of knowledge distillation (KD) for the head pose estimation problem. A student model trained by the proposed KD achieves results better than a teacher model, which is atypical for the response-based method. Our method consists of two stages. In the first stage, we trained the base neural network (NN), which has one regression head and four regression via classification (RvC) heads. We build the convolutional ensemble over the base NN using offsets of face bounding boxes over a regular grid. In the second stage, we perform KD from the convolutional ensemble in…
Unleashing the Potential of Mamba: Boosting a LiDAR 3D Sparse Detector by Using Cross-Model Knowledge Distillation
East China University of Science and Technology（华东理工大学）；University of Sydney（悉尼大学）；College of Computer Science and Artificial Intelligence, Fudan University（复旦大学计算机科学技术学院）
Socialized Detector Learning: Trajectory-Guided and Reciprocal Distillation for Heterogeneous Object Detectors
School of New Media and Communication, Tianjin University（天津大学新媒体与传播学院）；School of Artificial Intelligence, Tianjin University（天津大学人工智能学院）；School of Computer Science and Technology, Tianjin University（天津大学计算机科学与技术学院）；School of Sports Training, Tianjin University of Sport（天津体育学院运动训练学院）；Faculty of Information Engineering and Automation, Kunming University of Science and Technology（昆明理工大学信息工程与自动化学院）；School of Electronic Science, National University of Defense Technology（国防科技大学电子科学学院）；School of Automation, Southeast University（东南大学自动化学院）；School of Computer Science and Engineering, University of New South Wales（新南威尔士大学计算机科学与工程学院）
Reflect to Inform: Boosting Multimodal Reasoning via Information-Gain-Driven Verification
University of Science and Technology of China（中国科学技术大学）；Huawei Foundation Model Department（华为基础模型部）；The Chinese University of Hong Kong（香港中文大学）；Hong Kong University of Science and Technology（香港科技大学）
Object re-identification (re-id) aims to identify a specific object across times or camera views, with the person re-id and vehicle re-id as the most widely studied applications. Re-id is challenging because of the variations in viewpoints, (human) poses, and occlusions. Multi-shots of the same object can cover diverse viewpoints/poses and thus provide more comprehensive information. In this paper, we propose exploiting the multi-shots of the same identity to guide the feature learning of each individual image. Specifically, we design an Uncertainty-aware Multi-shot Teacher-Student (UMTS) Netw…
From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labels
Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；International Digital Economy Academy (IDEA)（国际数字经济研究院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
We introduce a simple yet effective distillation framework that is able to boost the vanilla ResNet-50 to 80%+ Top-1 accuracy on ImageNet without tricks. We construct such a framework through analyzing the problems in the existing classification system and simplify the base method ensemble knowledge distillation via discriminators by: (1) adopting the similarity loss and discriminator only on the final outputs and (2) using the average of softmax probabilities from all teacher ensembles as the stronger supervision. Intriguingly, three novel perspectives are presented for distillation: (1) weig…
GeoTeacher: Geometry-Guided Semi-Supervised 3D Object Detection
Fudan University（复旦大学）；Shanghai Innovation Institute（上海创新研究院）；Tongji University（同济大学）；Hong Kong University（香港大学）；Huazhong University of Science and technology（华中科技大学）
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Horizon Robotics（地平线机器人）
Unlike the conventional Knowledge Distillation (KD), Self-KD allows a network to learn knowledge from itself without any guidance from extra networks. This paper proposes to perform Self-KD from image Mixture (MixSKD), which integrates these two techniques into a unified framework. MixSKD mutually distills feature maps and probability distributions between the random pair of original images and their mixup images in a meaningful way. Therefore, it guides the network to learn cross-image knowledge by modelling supervisory signals from mixup images. Moreover, we construct a self-teacher network …
