The Devil is in Classification: A Simple Framework for Long-tail Object Detection and Instance Segmentation
NGS, National University of Singapore（新加坡国立大学）；ECE Department, National University of Singapore（新加坡国立大学电机与计算机工程系）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Salesforce Research Asia（Salesforce亚洲研究院）
Tao Wang, Yu Li, Bingyi Kang, Junnan Li, Junhao Liew, Sheng Tang, Steven Hoi, Jiashi Feng
备注：
LVIS 2019 challenge winner, performance significantly improved after challenge submission, accepted at ECCV 2020
Most existing object instance detection and segmentation models only work well on fairly balanced benchmarks where per-category training sample numbers are comparable, such as COCO. They tend to suffer performance drop on realistic datasets that are usually long-tailed. This work aims to study and address such open challenges. Specifically, we systematically investigate performance drop of the state-of-the-art two-stage instance segmentation model Mask R-CNN on the recent long-tail LVIS dataset, and unveil that a major cause is the inaccurate classification of object proposals. Based on such an observation, we first consider various techniques for improving long-tail classification performance which indeed enhance instance segmentation results. We then propose a simple calibration framework to more effectively alleviate classification head bias with a bi-level class balanced sampling approach. Without bells and whistles, it significantly boosts the performance of instance segmentation for tail classes on the recent LVIS dataset and our sampled COCO-LT dataset. Our analysis provides useful insights for solving long-tail instance detection and segmentation problems, and the straightforward \emph{SimCal} method can serve as a simple but strong baseline. With the method we have won the 2019 LVIS challenge. Codes and models are available at https://github.com/twangnh/SimCal.
Pursuing a more coherent scene understanding towards real-time vision applications, single-stage instance segmentation has recently gained popularity, achieving a simpler and more efficient design than its two-stage counterparts. Besides, its global mask representation often leads to superior accuracy to the two-stage Mask R-CNN which has been dominant thus far. Despite the promising advances in single-stage methods, finer delineation of instance boundaries still remains unexcavated. Indeed, boundary information provides a strong shape representation that can operate in synergy with the fully-…
NAVER Cloud（NAVER云）；KAIST（韩国科学技术院）；NAVER AI Lab（NAVER人工智能实验室）
In this paper, we introduce a novel learning scheme named weakly semi-supervised instance segmentation (WSSIS) with point labels for budget-efficient and high-performance instance segmentation. Namely, we consider a dataset setting consisting of a few fully-labeled images and a lot of point-labeled images. Motivated by the main challenge of semi-supervised approaches mainly derives from the trade-off between false-negative and false-positive instance proposals, we propose a method for WSSIS that can effectively leverage the budget-friendly point labels as a powerful weak supervision source to …
Devil is in Channels: Contrastive Single Domain Generalization for Medical Image Segmentation
Northwestern Polytechnical University（西北工业大学）；Ningbo Institute of Northwestern Polytechnical University（西北工业大学宁波研究院）；Research and Development Institute of Northwestern Polytechnical University in Shenzhen（西北工业大学深圳研究院）
The Devil is in the Statistics: Mitigating and Exploiting Statistics Difference for Generalizable Semi-supervised Medical Image Segmentation
State Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；National Institute of Healthcare Data Science, Nanjing University（南京大学健康医疗大数据国家研究院）；School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；School of Data and Computer Science, Shandong Women’s University（山东女子学院数据与计算机科学学院）
Cell Instance Segmentation: The Devil Is in the Boundaries
Department of Computer Science and Engineering, University of Notre Dame（计算机科学与工程系，诺丁汉大学）；Leibniz-Institut für Analytische Wissenschaften–ISAS–e.V.（莱比锡分析科学研究所）；School of Computing and Informatics, University of Louisiana at Lafayette（计算与信息学系，路易斯安那州立大学拉斐特分校）
Devil is in the Queries: Advancing Mask Transformers for Real-world Medical Image Segmentation and Out-of-Distribution Localization
Alibaba Group（阿里巴巴集团）；Peking University（北京大学）；Shengjing Hospital（盛京医院）；Guangdong Province People’s Hospital（广东省人民医院）；Peking University Changsha Institute for Computing and Digital Economy（北京大学长沙计算与数字经济研究院）
First Place Solution to the MLCAS 2025 GWFSS Challenge: The Devil is in the Detail and Minority
National Key Laboratory of Multispectral Information Intelligent Processing Technology（multispectral information intelligent processing technology national key laboratory）；School of Artificial Intelligence and Automation（人工智能与自动化学院）；Huazhong University of Science and Technology（华中科技大学）
Alibaba DAMO Academy（阿里巴巴达摩院）；The University of Adelaide（阿德莱德大学）
Recently, significant progress has been made on semantic segmentation. However, the success of supervised semantic segmentation typically relies on a large amount of labelled data, which is time-consuming and costly to obtain. Inspired by the success of semi-supervised learning methods in image classification, here we propose a simple yet effective semi-supervised learning framework for semantic segmentation. We demonstrate that the devil is in the details: a set of simple design and training techniques can collectively improve the performance of semi-supervised semantic segmentation significa…
University of California San Diego（加州大学圣迭戈分校）；Intel Labs（英特尔实验室）
Significant effort has been recently devoted to modeling visual relations. This has mostly addressed the design of architectures, typically by adding parameters and increasing model complexity. However, visual relation learning is a long-tailed problem, due to the combinatorial nature of joint reasoning about groups of objects. Increasing model complexity is, in general, ill-suited for long-tailed problems due to their tendency to overfit. In this paper, we explore an alternative hypothesis, denoted the Devil is in the Tails. Under this hypothesis, better performance is achieved by keeping the…
King’s College London（伦敦国王学院）；University of Liverpool（利物浦大学）；Imperial College London（帝国理工学院）
Major advancements have been made in the field of object detection and segmentation recently. However, when it comes to rare categories, the state-of-the-art methods fail to detect them, resulting in a significant performance gap between rare and frequent categories. In this paper, we identify that Sigmoid or Softmax functions used in deep detectors are a major reason for low performance and are sub-optimal for long-tailed detection and segmentation. To address this, we develop a Gumbel Optimized Loss (GOL), for long-tailed detection and segmentation. It aligns with the Gumbel distribution of …
Recent methods for long-tailed instance segmentation still struggle on rare object classes with few training data. We propose a simple yet effective method, Feature Augmentation and Sampling Adaptation (FASA), that addresses the data scarcity issue by augmenting the feature space especially for rare classes. Both the Feature Augmentation (FA) and feature sampling components are adaptive to the actual training status -- FA is informed by the feature mean and variance of observed real samples from past iterations, and we sample the generated virtual features in a loss-adapted manner to avoid ove…
National Tsing Hua University（台湾清华大学）；Virginia Tech（弗吉尼亚理工大学）；Aeolus Robotics
Long-tailed class distributions are prevalent among the practical applications of object detection and instance segmentation. Prior work in long-tail instance segmentation addresses the imbalance of losses between rare and frequent categories by reducing the penalty for a model incorrectly predicting a rare class label. We demonstrate that the rare categories are heavily suppressed by correct background predictions, which reduces the probability for all foreground categories with equal weight. Due to the relative infrequency of rare categories, this leads to an imbalance that biases towards pr…
The world is long-tailed. What does this mean for computer vision and visual recognition? The main two implications are (1) the number of categories we need to consider in applications can be very large, and (2) the number of training examples for most categories can be very small. Current visual recognition algorithms have achieved excellent classification accuracy. However, they require many training examples to reach peak performance, which suggests that long-tailed distributions will not be dealt with well. We analyze this question in the context of eBird, a large fine-grained classificati…
Instance segmentation is an active topic in computer vision that is usually solved by using supervised learning approaches over very large datasets composed of object level masks. Obtaining such a dataset for any new domain can be very expensive and time-consuming. In addition, models trained on certain annotated categories do not generalize well to unseen objects. The goal of this paper is to propose a method that can perform unsupervised discovery of long-tail categories in instance segmentation, through learning instance embeddings of masked regions. Leveraging rich relationship and hierarc…
Frequency-based Matcher for Long-tailed Semantic Segmentation
School of Computer Science, Beijing University of Posts and Telecommunications（北京邮电大学计算机学院）；School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；University of Technology Sydney（悉尼科技大学）
National University of Singapore（新加坡国立大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Salesforce Research Asia（Salesforce亚洲研究院）
Remarkable progress has been made in object instance detection and segmentation in recent years. However, existing state-of-the-art methods are mostly evaluated with fairly balanced and class-limited benchmarks, such as Microsoft COCO dataset [8]. In this report, we investigate the performance drop phenomenon of state-of-the-art two-stage instance segmentation models when processing extreme long-tail training data based on the LVIS [5] dataset, and find a major cause is the inaccurate classification of object proposals. Based on this observation, we propose to calibrate the prediction of class…
Federated Client-tailored Adapter for Medical Image Segmentation
Key Laboratory of Intelligent Computing and Signal Processing of Ministry of Education（教育部智能计算与信号处理重点实验室）；Anhui Provincial Key Laboratory of Security Artificial Intelligence（安徽省安全人工智能重点实验室）；School of Artificial Intelligence, Anhui University（安徽大学人工智能学院）；School of Internet, Anhui University（安徽大学互联网学院）；Anhui Provincial Key Laboratory of Multimodal Cognitive Computation（安徽省多模态认知计算重点实验室）；School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）；Department of Computer Science, The University of Hong Kong（香港大学计算机科学系）
The Chinese University of Hong Kong（香港中文大学）；Nanyang Technological University（南洋理工大学）；SenseTime Research（商汤科技研究院）；Shanghai AI Laboratory（上海人工智能实验室）；Zhejiang University（浙江大学）；University of Science and Technology of China（中国科学技术大学）
Instance segmentation has witnessed a remarkable progress on class-balanced benchmarks. However, they fail to perform as accurately in real-world scenarios, where the category distribution of objects naturally comes with a long tail. Instances of head classes dominate a long-tailed dataset and they serve as negative samples of tail categories. The overwhelming gradients of negative samples on tail classes lead to a biased learning process for classifiers. Consequently, objects of tail categories are more likely to be misclassified as backgrounds or head categories. To tackle this problem, we p…
SRMF: A Data Augmentation and Multimodal Fusion Approach for Long-Tail UHR Satellite Image Segmentation
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Haina Institute of Zhejiang University, Advanced Technology Institute, Zhejiang University（浙江大学海纳研究院、先进技术研究院）；Binjiang Research Institute of Zhejiang University（浙江大学滨江研究院）
The Chinese University of Hong Kong（香港中文大学）；Microsoft Research Asia（微软亚洲研究院）
In this paper, we study the problem of class imbalance in semantic segmentation. We first investigate and identify the main challenges of addressing this issue through pixel rebalance. Then a simple and yet effective region rebalance scheme is derived based on our analysis. In our solution, pixel features belonging to the same class are grouped into region features, and a rebalanced region classifier is applied via an auxiliary region rebalance branch during training. To verify the flexibility and effectiveness of our method, we apply the region rebalance module into various semantic segmentat…
ShanghaiTech University（上海科技大学）；Chinese University of Hong Kong（香港中文大学）；Shanghai Engineering Research Center of Intelligent Vision and Imaging（上海智能视觉与成像工程技术研究中心）
A thorough and holistic scene understanding is crucial for autonomous vehicles, where LiDAR semantic segmentation plays an indispensable role. However, most existing methods focus on the network design while neglecting the inherent difficulty, imbalanced data distribution in the realistic dataset (also named long-tailed distribution), which narrows down the capability of state-of-the-art methods. In this paper, we propose an input-output balanced framework to handle the issue of long-tailed distribution. Specifically, for the input space, we synthesize these tailed instances from mesh models a…
One fundamental challenge in building an instance segmentation model for a large number of classes in complex scenes is the lack of training examples, especially for rare objects. In this paper, we explore the possibility to increase the training examples without laborious data collection and annotation. We find that an abundance of instance segments can potentially be obtained freely from object-centric images, according to two insights: (i) an object-centric image usually contains one salient object in a simple background; (ii) objects from the same class often share similar appearances or s…
Nanjing University（南京大学）；MEGVII Technology（旷视科技）；Nanjing University of Science and Technology（南京理工大学）
Long-tailed instance segmentation is a challenging task due to the extreme imbalance of training samples among classes. It causes severe biases of the head classes (with majority samples) against the tailed ones. This renders "how to appropriately define and alleviate the bias" one of the most important issues. Prior works mainly use label distribution or mean score information to indicate a coarse-grained bias. In this paper, we explore to excavate the confusion matrix, which carries the fine-grained misclassification details, to relieve the pairwise biases, generalizing the coarse one. To th…
The Devil is in Fine-tuning and Long-tailed Problems:A New Benchmark for Scene Text Detection
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；VCIP & TMCC & DISSec, College of Computer Science, Nankai University（南开大学计算机学院）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）
Class-Imbalanced Semi-Supervised Learning for Large-Scale Point Cloud Semantic Segmentation via Decoupling Optimization
Shanghai University（上海大学）；Shanghai Engineering Research Center of Motion Picture Special Effects（上海电影特效工程技术研究中心）；East China Normal University（华东师范大学）；Tencent Youtu Lab（腾讯优图实验室）；Beihang University（北京航空航天大学）
FZI Research Center for Information Technology（FZI信息技术研究中心）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）
In this paper we introduce a novel way to predict semantic information from sparse, single-shot LiDAR measurements in the context of autonomous driving. In particular, we fuse learned features from complementary representations. The approach is aimed specifically at improving the semantic segmentation of top-view grid maps. Towards this goal the 3D LiDAR point cloud is projected onto two orthogonal 2D representations. For each representation a tailored deep learning architecture is developed to effectively extract semantic information which are fused by a superordinate deep neural network. The…
State University of New York at Buffalo（纽约州立大学布法罗分校）；Tianjin University（天津大学）；Horizon Robotics, Inc.（地平线机器人科技有限公司）
Despite the previous success of object analysis, detecting and segmenting a large number of object categories with a long-tailed data distribution remains a challenging problem and is less investigated. For a large-vocabulary classifier, the chance of obtaining noisy logits is much higher, which can easily lead to a wrong recognition. In this paper, we exploit prior knowledge of the relations among object categories to cluster fine-grained classes into coarser parent classes, and construct a classification tree that is responsible for parsing an object instance into a fine-grained category via…
Generalizable Multiscale Segmentation of Heterogeneous Map Collections
EPFL, Swiss Federal Institute of Technology in Lausanne. Digital Humanities Laboratory (DHLAB), School of Computer（瑞士联邦理工学院洛桑分校，数字人文实验室）；EPFL. Laboratory of Urban Sociology (LASUR), School of Architecture, Civil（瑞士联邦理工学院洛桑分校，城市社会学实验室）
This report introduces the technical details of the team FuXi-Fresher for LVIS Challenge 2021. Our method focuses on the problem in following two aspects: the long-tail distribution and the segmentation quality of mask and boundary. Based on the advanced HTC instance segmentation algorithm, we connect transformer backbone(Swin-L) through composite connections inspired by CBNetv2 to enhance the baseline results. To alleviate the problem of long-tail distribution, we design a Distribution Balanced method which includes dataset balanced and loss function balaced modules. Further, we use a Mask an…
Mix-modal Federated Learning for MRI Image Segmentation
School of Artificial Intelligence（人工智能学院）；School of Computer Science and Technology（计算机科学与技术学院）；State Key Laboratory of Opto-Electronic Information Acquisition and Protection Technology（光电信息采集与防护技术国家重点实验室）；Anhui Provincial Key Laboratory of Security Artificial Intelligence（安徽省安全人工智能重点实验室）；Anhui Provincial Key Laboratory of Multimodal Cognitive Computation（安徽省多模态认知计算重点实验室）
The Hong Kong Polytechnic University（香港理工大学）；Dalian University of Technology（大连理工大学）
Domain adaptive semantic segmentation aims to learn a model with the supervision of source domain data, and produce satisfactory dense predictions on unlabeled target domain. One popular solution to this challenging task is self-training, which selects high-scoring predictions on target samples as pseudo labels for training. However, the produced pseudo labels often contain much noise because the model is biased to source domain as well as majority categories. To address the above issues, we propose to directly explore the intrinsic pixel distributions of target domain data, instead of heavily…
University College London（伦敦大学学院）；Google Inc.（谷歌公司）
Many machine vision applications, such as semantic segmentation and depth prediction, require predictions for every pixel of the input image. Models for such problems usually consist of encoders which decrease spatial resolution while learning a high-dimensional representation, followed by decoders who recover the original input resolution and result in low-dimensional predictions. While encoders have been studied rigorously, relatively few studies address the decoder side. This paper presents an extensive comparison of a variety of decoders for a variety of pixel-wise tasks ranging from class…
The Ohio State University（俄亥俄州立大学）；Google Research（谷歌研究院）
Vanilla models for object detection and instance segmentation suffer from the heavy bias toward detecting frequent objects in the long-tailed setting. Existing methods address this issue mostly during training, e.g., by re-sampling or re-weighting. In this paper, we investigate a largely overlooked approach -- post-processing calibration of confidence scores. We propose NorCal, Normalized Calibration for long-tailed object detection and instance segmentation, a simple and straightforward recipe that reweighs the predicted scores of each class by its training sample size. We show that separatel…
Long-tailed learning aims to tackle the crucial challenge that head classes dominate the training procedure under severe class imbalance in real-world scenarios. However, little attention has been given to how to quantify the dominance severity of head classes in the representation space. Motivated by this, we generalize the cosine-based classifiers to a von Mises-Fisher (vMF) mixture model, denoted as vMF classifier, which enables to quantitatively measure representation quality upon the hyper-sphere space via calculating distribution overlap coefficient. To our knowledge, this is the first w…
We propose an approach to semantic (image) segmentation that reduces the computational costs by a factor of 25 with limited impact on the quality of results. Semantic segmentation has a number of practical applications, and for most such applications the computational costs are critical. The method follows a typical two-column network structure, where one column accepts an input image, while the other accepts a half-resolution version of that image. By identifying specific regions in the full-resolution image that can be safely ignored, as well as carefully tailoring the network structure, we …
The University of Hong Kong（香港大学）；Zhejiang University（浙江大学）
While self-training has advanced semi-supervised semantic segmentation, it severely suffers from the long-tailed class distribution on real-world semantic segmentation datasets that make the pseudo-labeled data bias toward majority classes. In this paper, we present a simple and yet effective Distribution Alignment and Random Sampling (DARS) method to produce unbiased pseudo labels that match the true class distribution estimated from the labeled data. Besides, we also contribute a progressive data augmentation and labeling strategy to facilitate model training with pseudo-labeled data. Experi…
Universidad de los Andes（安第斯大学）；University of California, Berkeley（加州大学伯克利分校）
We aim to detect all instances of a category in an image and, for each instance, mark the pixels that belong to it. We call this task Simultaneous Detection and Segmentation (SDS). Unlike classical bounding box detection, SDS requires a segmentation and not just a box. Unlike classical semantic segmentation, we require individual object instances. We build on recent work that uses convolutional neural networks to classify category-independent region proposals (R-CNN [16]), introducing a novel architecture tailored for SDS. We then use category-specific, top- down figure-ground predictions to r…
Facebook AI Research (FAIR)（Facebook人工智能研究院（FAIR））
Progress on object detection is enabled by datasets that focus the research community's attention on open challenges. This process led us from simple images to complex scenes and from bounding boxes to segmentation masks. In this work, we introduce LVIS (pronounced `el-vis'): a new dataset for Large Vocabulary Instance Segmentation. We plan to collect ~2 million high-quality instance segmentation masks for over 1000 entry-level object categories in 164k images. Due to the Zipfian distribution of categories in natural images, LVIS naturally has a long tail of categories with few training sample…
