In many real-world applications, the frequency distribution of class labels for training data can exhibit a long-tailed distribution, which challenges traditional approaches of training deep neural networks that require heavy amounts of balanced data. Gathering and labeling data to balance out the class label distribution can be both costly and time-consuming. Many existing solutions that enable ensemble learning, re-balancing strategies, or fine-tuning applied to deep neural networks are limited by the inert problem of few class samples across a subset of classes. Recently, vision-language models like CLIP have been observed as effective solutions to zero-shot or few-shot learning by grasping a similarity between vision and language features for image and text pairs. Considering that large pre-trained vision-language models may contain valuable side textual information for minor classes, we propose to leverage text supervision to tackle the challenge of long-tailed learning. Concretely, we propose a novel text-guided mixup technique that takes advantage of the semantic relations between classes recognized by the pre-trained text encoder to help alleviate the long-tailed problem. Our empirical study on benchmark long-tailed tasks demonstrates the effectiveness of our proposal with a theoretical guarantee. Our code is available at https://github.com/rsamf/text-guided-mixup.
Label-Occurrence-Balanced Mixup for Long-tailed Recognition
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Inner Mongolia Key Laboratory of Molecular Biology on Featured Plants(内蒙古特色植物分子生物学重点实验室)
Saliency-guided and Patch-based Mixup for Long-tailed Skin Cancer Image Classification
Southern University of Science and Technology(南方科技大学);University of British Columbia(不列颠哥伦比亚大学);The University of Hong Kong(香港大学);Jiaxing Research Institute, Southern University of Science and Technology(南方科技大学嘉兴研究院)
Concept Drift and Long-Tailed Distribution in Fine-Grained Visual Categorization: Benchmark and Method
School of Electronic Information and Communications, Huazhong University of Science and Technology(华中科技大学电子信息与通信学院);Alibaba Group, DEEPROUTE.AI(阿里巴巴集团DEEPROUTE.AI);Mohamed bin Zayed University of AI(穆罕默德·本·扎耶德人工智能大学);Australian National University(澳大利亚国立大学);Linkoping University(林雪平大学);University of Chinese Academy of Sciences(中国科学院大学)
EndoOOD: Uncertainty-aware Out-of-distribution Detection in Capsule Endoscopy Diagnosis
The Chinese University of Hong Kong(香港中文大学);Shenzhen Research Institute, The Chinese University of Hong Kong(香港中文大学深圳研究院);University College London(伦敦大学学院);National University of Singapore(新加坡国立大学)