Real-world visual recognition requires handling the extreme sample imbalance in large-scale long-tailed data. We propose a "divide&conquer" strategy for the challenging LVIS task: divide the whole data into balanced parts and then apply incremental learning to conquer each one. This derives a novel learning paradigm: class-incremental few-shot learning, which is especially effective for the challenge evolving over time: 1) the class imbalance among the old-class knowledge review and 2) the few-shot data in new-class learning. We call our approach Learning to Segment the Tail (LST). In particular, we design an instance-level balanced replay scheme, which is a memory-efficient approximation to balance the instance-level samples from the old-class images. We also propose to use a meta-module for new-class learning, where the module parameters are shared across incremental phases, gaining the learning-to-learn knowledge incrementally, from the data-rich head to the data-poor tail. We empirically show that: at the expense of a little sacrifice of head-class forgetting, we can gain a significant 8.3% AP improvement for the tail classes with less than 10 instances, achieving an overall 2.0% AP boost for the whole 1,230 classes.
One fundamental challenge in building an instance segmentation model for a large number of classes in complex scenes is the lack of training examples, especially for rare objects. In this paper, we explore the possibility to increase the training examples without laborious data collection and annotation. We find that an abundance of instance segments can potentially be obtained freely from object-centric images, according to two insights: (i) an object-centric image usually contains one salient object in a simple background; (ii) objects from the same class often share similar appearances or s…
NGS, National University of Singapore（新加坡国立大学）；ECE Department, National University of Singapore（新加坡国立大学电机与计算机工程系）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Salesforce Research Asia（Salesforce亚洲研究院）
Most existing object instance detection and segmentation models only work well on fairly balanced benchmarks where per-category training sample numbers are comparable, such as COCO. They tend to suffer performance drop on realistic datasets that are usually long-tailed. This work aims to study and address such open challenges. Specifically, we systematically investigate performance drop of the state-of-the-art two-stage instance segmentation model Mask R-CNN on the recent long-tail LVIS dataset, and unveil that a major cause is the inaccurate classification of object proposals. Based on such a…
National Tsing Hua University（台湾清华大学）；Virginia Tech（弗吉尼亚理工大学）；Aeolus Robotics
Long-tailed class distributions are prevalent among the practical applications of object detection and instance segmentation. Prior work in long-tail instance segmentation addresses the imbalance of losses between rare and frequent categories by reducing the penalty for a model incorrectly predicting a rare class label. We demonstrate that the rare categories are heavily suppressed by correct background predictions, which reduces the probability for all foreground categories with equal weight. Due to the relative infrequency of rare categories, this leads to an imbalance that biases towards pr…
Instance segmentation is an active topic in computer vision that is usually solved by using supervised learning approaches over very large datasets composed of object level masks. Obtaining such a dataset for any new domain can be very expensive and time-consuming. In addition, models trained on certain annotated categories do not generalize well to unseen objects. The goal of this paper is to propose a method that can perform unsupervised discovery of long-tail categories in instance segmentation, through learning instance embeddings of masked regions. Leveraging rich relationship and hierarc…
National University of Singapore（新加坡国立大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Salesforce Research Asia（Salesforce亚洲研究院）
Remarkable progress has been made in object instance detection and segmentation in recent years. However, existing state-of-the-art methods are mostly evaluated with fairly balanced and class-limited benchmarks, such as Microsoft COCO dataset [8]. In this report, we investigate the performance drop phenomenon of state-of-the-art two-stage instance segmentation models when processing extreme long-tail training data based on the LVIS [5] dataset, and find a major cause is the inaccurate classification of object proposals. Based on this observation, we propose to calibrate the prediction of class…
The Chinese University of Hong Kong（香港中文大学）；Nanyang Technological University（南洋理工大学）；SenseTime Research（商汤科技研究院）；Shanghai AI Laboratory（上海人工智能实验室）；Zhejiang University（浙江大学）；University of Science and Technology of China（中国科学技术大学）
Instance segmentation has witnessed a remarkable progress on class-balanced benchmarks. However, they fail to perform as accurately in real-world scenarios, where the category distribution of objects naturally comes with a long tail. Instances of head classes dominate a long-tailed dataset and they serve as negative samples of tail categories. The overwhelming gradients of negative samples on tail classes lead to a biased learning process for classifiers. Consequently, objects of tail categories are more likely to be misclassified as backgrounds or head categories. To tackle this problem, we p…
SRMF: A Data Augmentation and Multimodal Fusion Approach for Long-Tail UHR Satellite Image Segmentation
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Haina Institute of Zhejiang University, Advanced Technology Institute, Zhejiang University（浙江大学海纳研究院、先进技术研究院）；Binjiang Research Institute of Zhejiang University（浙江大学滨江研究院）
Nanjing University（南京大学）；MEGVII Technology（旷视科技）；Nanjing University of Science and Technology（南京理工大学）
Long-tailed instance segmentation is a challenging task due to the extreme imbalance of training samples among classes. It causes severe biases of the head classes (with majority samples) against the tailed ones. This renders "how to appropriately define and alleviate the bias" one of the most important issues. Prior works mainly use label distribution or mean score information to indicate a coarse-grained bias. In this paper, we explore to excavate the confusion matrix, which carries the fine-grained misclassification details, to relieve the pairwise biases, generalizing the coarse one. To th…
Divide, Weight, and Route: Difficulty-Aware Optimization with Dynamic Expert Fusion for Long-tailed Recognition
the MIIT Key Laboratory of Pattern Analysis and Machine Intelligence（信息模式分析与机器智能国家重点实验室）；Collaborative Innovation Center of Novel Software Technology and Industrialization（新型软件技术与产业化协同创新中心）；College of Computer Science and Technology（计算机科学与技术学院）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）
Synthetic training for long-tail haemorrhagic lesion segmentation in data-scarce settings
Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU)（埃尔朗根-纽伦堡弗里德里希-亚历山大大学（FAU））；Otto von Guericke University Magdeburg（马格德堡奥托·冯·格里克大学）；German Centre for Neurodegenerative Diseases (DZNE)（德国神经退行性疾病中心（DZNE））
Automated Sperm Morphology Analysis Based on Instance-Aware Part Segmentation
Robotics Institute, University of Toronto（多伦多大学机器人研究所）；School of Mechanical Engineering, Dalian University of Technology（大连理工大学机械工程学院）；School Of Artificial Intelligence, Henan University（河南大学人工智能学院）；CReATe Fertility Center（CReATe生殖中心）；School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）理工学院）
Self-supervised learning holds promise in leveraging large amounts of unlabeled data, however much of its progress has thus far been limited to highly curated pre-training data such as ImageNet. We explore the effects of contrastive learning from larger, less-curated image datasets such as YFCC, and find there is indeed a large difference in the resulting representation quality. We hypothesize that this curation gap is due to a shift in the distribution of image classes -- which is more diverse and heavy-tailed -- resulting in less relevant negative samples to learn from. We test this hypothes…
State University of New York at Buffalo（纽约州立大学布法罗分校）；Tianjin University（天津大学）；Horizon Robotics, Inc.（地平线机器人科技有限公司）
Despite the previous success of object analysis, detecting and segmenting a large number of object categories with a long-tailed data distribution remains a challenging problem and is less investigated. For a large-vocabulary classifier, the chance of obtaining noisy logits is much higher, which can easily lead to a wrong recognition. In this paper, we exploit prior knowledge of the relations among object categories to cluster fine-grained classes into coarser parent classes, and construct a classification tree that is responsible for parsing an object instance into a fine-grained category via…
Class-Imbalanced Semi-Supervised Learning for Large-Scale Point Cloud Semantic Segmentation via Decoupling Optimization
Shanghai University（上海大学）；Shanghai Engineering Research Center of Motion Picture Special Effects（上海电影特效工程技术研究中心）；East China Normal University（华东师范大学）；Tencent Youtu Lab（腾讯优图实验室）；Beihang University（北京航空航天大学）
