Deep learning has achieved remarkable progress for visual recognition on large-scale balanced datasets but still performs poorly on real-world long-tailed data. Previous methods often adopt class re-balanced training strategies to effectively alleviate the imbalance issue, but might be a risk of over-fitting tail classes. The recent decoupling method overcomes over-fitting issues by using a multi-stage training scheme, yet, it is still incapable of capturing tail class information in the feature learning stage. In this paper, we show that soft label can serve as a powerful solution to incorporate label correlation into a multi-stage training scheme for long-tailed recognition. The intrinsic relation between classes embodied by soft labels turns out to be helpful for long-tailed recognition by transferring knowledge from head to tail classes.
Specifically, we propose a conceptually simple yet particularly effective multi-stage training scheme, termed as Self Supervised to Distillation (SSD). This scheme is composed of two parts. First, we introduce a self-distillation framework for long-tailed recognition, which can mine the label relation automatically. Second, we present a new distillation label generation module guided by self-supervision. The distilled labels integrate information from both label and data domains that can model long-tailed distribution effectively. We conduct extensive experiments and our method achieves the state-of-the-art results on three long-tailed recognition benchmarks: ImageNet-LT, CIFAR100-LT and iNaturalist 2018. Our SSD outperforms the strong LWS baseline by from $2.7\%$ to $4.5\%$ on various datasets. The code is available at https://github.com/MCG-NJU/SSD-LT.
Balanced Knowledge Distillation for Long-tailed Learning
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Nanjing University of Science and Technology（南京理工大学）
Relational Subsets Knowledge Distillation for Long-tailed Retinal Diseases Recognition
Airdoc LLC（鹰瞳科技）；Monash University（莫纳什大学）；Harbin Engineering University（哈尔滨工程大学）；The University of Sydney（悉尼大学）；Inception Institute of Artificial Intelligence（起源人工智能研究院）
Learning From Multiple Experts: Self-paced Knowledge Distillation for Long-tailed Classification
Tsinghua University（清华大学）；Beijing National Research Center for Information Science and Technology (BNRist)（北京信息科学与技术国家研究中心）；Aberystwyth University（阿伯里斯特威斯大学）
Learn from Balance: Rectifying Knowledge Transfer for Long-Tailed Scenarios
School of Computer Science and Technology, Southwest University of Science and Technology（西南科技大学计算机科学与技术学院）；School of Information Science and Technology, Great Bay University（大湾区大学信息科学与技术学院）；Graduate School of Science and Engineering, Hosei University（法政大学理工学研究院）
Distilling Virtual Examples for Long-tailed Recognition
State Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机科学与技术系（注：此处原机构为南京大学，具体实验室为 Novel Software Technology 重点实验室，按要求提取到大学级别））；School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与技术学院）
FoPro-KD: Fourier Prompted Effective Knowledge Distillation for Long-Tailed Medical Image Recognition
The Hong Kong University of Science and Technology（香港科技大学）；University of Girona（赫罗纳大学）；HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute（港科大深港协同创新研究院）
CEKD:Cross Ensemble Knowledge Distillation for Augmented Fine-grained Data
Hangzhou Dianzi University（杭州电子科技大学）；University of Sydney（悉尼大学）；UBTECH Sydney AI Centre（优必选悉尼人工智能中心）；Australian Centre for Field Robotics（澳大利亚现场机器人研究中心）
Hefei University of Technology（合肥工业大学）；Jianghuai Advance Technology Center（江淮先进技术中心）；Anhui Provincial Key Laboratory of Humanoid Robots（安徽省人形机器人重点实验室）；Kaiyang Laboratory, Chery（凯阳实验室，中国奇瑞）；Zhejiang University of Technology（浙江工业大学）
JointViT: Modeling Oxygen Saturation Levels with Joint Supervision on Long-Tailed OCTA
Flinders University（弗林德斯大学）；The Australian National University（澳大利亚国立大学）；The University of Adelaide（阿德莱德大学）；Guangdong Technion - Israel Institute of Technology（广东以色列理工学院）；The University of Sydney（悉尼大学）
Zhejiang University（浙江大学）；Hangzhou City University（浙大城市学院）；ZJU-Bangsun Joint Research Center（浙江大学-邦盛联合研究中心）；Shanghai Institute for Advanced Study of Zhejiang University（浙江大学上海高等研究院）
Differentiable Feature Aggregation Search for Knowledge Distillation
Peking University（北京大学）；Megvii (Face++) Technology Inc（旷视科技（Face++））；National Engineering Laboratory for Big Data Analysis and Applications（大数据分析与应用技术国家工程实验室）；DiDi AI Labs（滴滴人工智能实验室）
Long-Tailed Recognition on Binary Networks by Calibrating A Pre-trained Model
Minds and Company（明兹公司）；Upstage AI（上阶段人工智能公司）；Gwangju Institute of Science and Technology (GIST)（光州科学技术院）；Alsemy（阿尔塞米公司）；Seoul National University（首尔大学）
Beyond Modality Collapse: Representations Blending for Multimodal Dataset Distillation
Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore（前沿人工智能研究中心，科技研究局，新加坡）；Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore（高性能计算研究所，科技研究局，新加坡）；National University of Singapore, Singapore（新加坡国立大学）；Zhejiang University, China（浙江大学）
BAS-OPD: Budget-Aware Selective On-Policy Self-Distillation for Fine-Grained Multimodal Perception
HFIPS, Chinese Academy of Sciences（中国科学院合肥物质科学研究院）；University of Science and Technology of China（中国科学技术大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
Not All Tokens Matter Equally: Dynamic In-context Vector Distillation with Decisive-Token Supervision for Long-form Medical Report Generation
UNSW Sydney（新南威尔士大学悉尼分校）；University of Technology Sydney（技术大学悉尼分校）；School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；Stanford University（斯坦福大学）；Shanghai Jiao Tong University（上海交通大学）
Soft Label Pruning and Quantization for Large-Scale Dataset Distillation
CFAR, Agency for Science, Technology and Research, Singapore（新加坡科技研究局应用科学与技术研究院）；IHPC, Agency for Science, Technology and Research, Singapore（新加坡科技研究局信息与通信技术研究院）；National University of Singapore（新加坡国立大学）
