Class-Imbalanced Semi-Supervised Learning for Large-Scale Point Cloud Semantic Segmentation via Decoupling Optimization
Shanghai University（上海大学）；Shanghai Engineering Research Center of Motion Picture Special Effects（上海电影特效工程技术研究中心）；East China Normal University（华东师范大学）；Tencent Youtu Lab（腾讯优图实验室）；Beihang University（北京航空航天大学）
Semi-supervised learning (SSL), thanks to the significant reduction of data annotation costs, has been an active research topic for large-scale 3D scene understanding. However, the existing SSL-based methods suffer from severe training bias, mainly due to class imbalance and long-tail distributions of the point cloud data. As a result, they lead to a biased prediction for the tail class segmentation. In this paper, we introduce a new decoupling optimization framework, which disentangles feature representation learning and classifier in an alternative optimization manner to shift the bias decision boundary effectively. In particular, we first employ two-round pseudo-label generation to select unlabeled points across head-to-tail classes. We further introduce multi-class imbalanced focus loss to adaptively pay more attention to feature learning across head-to-tail classes. We fix the backbone parameters after feature learning and retrain the classifier using ground-truth points to update its parameters. Extensive experiments demonstrate the effectiveness of our method outperforming previous state-of-the-art methods on both indoor and outdoor 3D point cloud datasets (i.e., S3DIS, ScanNet-V2, Semantic3D, and SemanticKITTI) using 1% and 1pt evaluation.
To overcome the data-hungry challenge, we have proposed a semi-supervised contrastive learning framework for the task of class-imbalanced semantic segmentation. First and foremost, to make the model operate in a semi-supervised manner, we proposed the confidence-level-based contrastive learning to achieve instance discrimination in an explicit manner, and make the low-confidence low-quality features align with the high-confidence counterparts. Moreover, to tackle the problem of class imbalance in crack segmentation and road components extraction, we proposed the data imbalance loss to replace …
Max-Delbrueck-Center for Molecular Medicine in the Helmholtz Association (MDC)（亥姆霍兹联合会马克斯·德尔布吕克分子医学中心）；Humboldt-Universität zu Berlin（柏林洪堡大学）；Case Western Reserve University（凯斯西储大学）；Lausanne University Hospital（洛桑大学医院）；University of Bern（伯尔尼大学）；Charité University Medicine（夏里特医科大学）
We describe here the panoptic segmentation method we devised for our participation in the CoNIC: Colon Nuclei Identification and Counting Challenge at ISBI 2022. Key features of our method are a weighted loss specifically engineered for semantic segmentation of highly imbalanced cell types, and a state-of-the art nuclei instance segmentation model, which we combine in a Hovernet-like architecture.
A recent study has shown a phenomenon called neural collapse in that the within-class means of features and the classifier weight vectors converge to the vertices of a simplex equiangular tight frame at the terminal phase of training for classification. In this paper, we explore the corresponding structures of the last-layer feature centers and classifiers in semantic segmentation. Based on our empirical and theoretical analysis, we point out that semantic segmentation naturally brings contextual correlation and imbalanced distribution among classes, which breaks the equiangular and maximally …
University of Arkansas（阿肯色大学）；Carnegie Mellon University（卡内基梅隆大学）
Medical image segmentation has played an important role in medical analysis and widely developed for many clinical applications. Deep learning-based approaches have achieved high performance in semantic segmentation but they are limited to pixel-wise setting and imbalanced classes data problem. In this paper, we tackle those limitations by developing a new deep learning-based model which takes into account both higher feature level i.e. region inside contour, intermediate feature level i.e. offset curves around the contour and lower feature level i.e. contour. Our proposed Offset Curves (OsC) …
School of Biological Science and Medical Engineering, Beihang University（北京航空航天大学生物科学与医学工程学院）；School of Computer Science and Technology, Xidian University（西安电子科技大学计算机科学与技术学院）；Hefei Innovation Research Institute, Beihang University（北京航空航天大学合肥创新研究院）；Beijing Advanced Innovation Centre for Biomedical Engineering, Beihang University（北京航空航天大学生物医学工程高精尖创新中心）；Beijing Advanced Innovation Centre for Big Data-Based Precision Medicine, Beihang University（北京航空航天大学大数据精准医疗高精尖创新中心）
Consistency training has proven to be an advanced semi-supervised framework and achieved promising results in medical image segmentation tasks through enforcing an invariance of the predictions over different views of the inputs. However, with the iterative updating of model parameters, the models would tend to reach a coupled state and eventually lose the ability to exploit unlabeled data. To address the issue, we present a novel semi-supervised segmentation model based on parameter decoupling strategy to encourage consistent predictions from diverse views. Specifically, we first adopt a two-…
Surgical-DeSAM: Decoupling SAM for Instrument Segmentation in Robotic Surgery
Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS)（Wellcome/EPSRC介入与外科科学中心（WEISS））；University College London（伦敦大学学院）；Dept. of Computer Science, University College London（伦敦大学学院计算机科学系）
Boston Children's Hospital（波士顿儿童医院）；Harvard Medical School（哈佛医学院）；Northeastern University（东北大学）
Fully convolutional deep neural networks have been asserted to be fast and precise frameworks with great potential in image segmentation. One of the major challenges in training such networks raises when data is unbalanced, which is common in many medical imaging applications such as lesion segmentation where lesion class voxels are often much lower in numbers than non-lesion voxels. A trained network with unbalanced data may make predictions with high precision and low recall, being severely biased towards the non-lesion class which is particularly undesired in most medical applications where…
South China University of Technology（华南理工大学）；Nanyang Technological University（南洋理工大学）
Data augmentation is vital for deep learning neural networks. By providing massive training samples, it helps to improve the generalization ability of the model. Weakly supervised semantic segmentation (WSSS) is a challenging problem that has been deeply studied in recent years, conventional data augmentation approaches for WSSS usually employ geometrical transformations, random cropping and color jittering. However, merely increasing the same contextual semantic data does not bring much gain to the networks to distinguish the objects, e.g., the correct image-level classification of "aeroplane…
Wuhan University（武汉大学）；MPI for Informatics（马克斯·普朗克信息学研究所）
Zero-shot semantic segmentation (ZS3) aims to segment the novel categories that have not been seen in the training. Existing works formulate ZS3 as a pixel-level zeroshot classification problem, and transfer semantic knowledge from seen classes to unseen ones with the help of language models pre-trained only with texts. While simple, the pixel-level ZS3 formulation shows the limited capability to integrate vision-language models that are often pre-trained with image-text pairs and currently demonstrate great potential for vision tasks. Inspired by the observation that humans often perform segm…
Deep Distance Map Regression Network with Shape-aware Loss for Imbalanced Medical Image Segmentation
Beijing Institute of Technology（北京理工大学）；School of Computer Science（计算机学院）；Beijing Lab of Intelligent Information Technology（北京智能信息技术实验室）；Chinese Academy of Medical Sciences（北京协和医学院（中国医学科学院））
Feature Decoupling-Recycling Network for Fast Interactive Segmentation
University of Science and Technology of China（中国科学技术大学）；Xiaohongshu Inc（小红书公司）；Kuaishou Technology（快手科技）；Dartmouth College（达特茅斯学院）；Harbin Institute of Technology（哈尔滨工业大学）
Decoupling Wavelet Sub-bands for Single Source Domain Generalization in Fundus Image Segmentation
Indian Statistical Institute（印度统计研究所）；University of Oxford（牛津大学）；Radcliffe Department of Medicine, University of Oxford（牛津大学拉德克利夫医学系）；Department of Engineering Science, University of Oxford（牛津大学工程科学系）
3D semantic segmentation (3DSS) is an essential process in the creation of a safe autonomous driving system. However, deep learning models for 3D semantic segmentation often suffer from the class imbalance problem and out-of-distribution (OOD) data. In this study, we explore how the class imbalance problem affects 3DSS performance and whether the model can detect the category prediction correctness, or whether data is ID (in-distribution) or OOD. For these purposes, we conduct two experiments using three representative 3DSS models and five trust scoring methods, and conduct both a confusion an…
Harmonizing output imbalance for defect segmentation on extremely-imbalanced photovoltaic module cells images
Shenzhen University（深圳大学）；Guangdong Artificial Intelligence and Digital Economy Laboratory (Shenzhen)（广东人工智能与数字经济实验室（深圳））；Shenzhen Institute of Technology（深圳技术学院）
Separate and Conquer: Decoupling Co-occurrence via Decomposition and Representation for Weakly Supervised Semantic Segmentation
Academy for Engineering and Technology, Fudan University（复旦大学工程技术学院）；Digital Medical Research Center, School of Basic Medical Sciences, Fudan University（复旦大学基础医学院数字医学研究中心）；Shandong Computer Science Center (National Supercomputer Center in Jinan)（山东计算中心（国家济南超级计算中心））
In this paper, we propose mean squared error (MSE) loss with outlying label for class imbalanced classification. Cross entropy (CE) loss, which is widely used for image recognition, is learned so that the probability value of true class is closer to one by back propagation. However, for imbalanced datasets, the learning is insufficient for the classes with a small number of samples. Therefore, we propose a novel classification method using the MSE loss that can be learned the relationships of all classes no matter which image is input. Unlike CE loss, MSE loss is possible to equalize the numbe…
Pseudo-Label Quality Decoupling and Correction for Semi-Supervised Instance Segmentation
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, China（中国教育部多媒体可信感知与高效计算重点实验室，厦门大学）；Tencent Youtu Lab（腾讯云视频实验室）
Mix-modal Federated Learning for MRI Image Segmentation
School of Artificial Intelligence（人工智能学院）；School of Computer Science and Technology（计算机科学与技术学院）；State Key Laboratory of Opto-Electronic Information Acquisition and Protection Technology（光电信息采集与防护技术国家重点实验室）；Anhui Provincial Key Laboratory of Security Artificial Intelligence（安徽省安全人工智能重点实验室）；Anhui Provincial Key Laboratory of Multimodal Cognitive Computation（安徽省多模态认知计算重点实验室）
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Dalian University of Technology（大连理工大学）；Nanjing University（南京大学）；Lingnan University（岭南大学）
Most of the existing object detection methods generate poor glass detection results, due to the fact that the transparent glass shares the same appearance with arbitrary objects behind it in an image. Different from traditional deep learning-based wisdoms that simply use the object boundary as auxiliary supervision, we exploit label decoupling to decompose the original labeled ground-truth (GT) map into an interior-diffusion map and a boundary-diffusion map. The GT map in collaboration with the two newly generated maps breaks the imbalanced distribution of the object boundary, leading to impro…
Visual Information Laboratory, University of Bristol, UK（布里斯托尔大学视觉信息实验室）
As a data-driven method, the performance of deep convolutional neural networks (CNN) relies heavily on training data. The prediction results of traditional networks give a bias toward larger classes, which tend to be the background in the semantic segmentation task. This becomes a major problem for fault detection, where the targets appear very small on the images and vary in both types and sizes. In this paper we propose a new network architecture, DefectNet, that offers multi-class (including but not limited to) defect detection on highly-imbalanced datasets. DefectNet consists of two parall…
Disentangling Sampling from Training Budget in Class-Imbalanced CT Body Composition Segmentation
Amsterdam University Medical Center（阿姆斯特丹大学医学中心）；University of Amsterdam（阿姆斯特丹大学）；Informatics Institute, Faculty of Science（信息学院，科学学院）；Department of Radiology（放射科）；Mayo Clinic Rochester（罗切斯特梅奥诊所）
Nanjing University（南京大学）；Shanghai AI Laboratory（上海人工智能实验室）；The University of Hong Kong（香港大学）；NVIDIA（英伟达）；Caltech（加州理工学院）
Panoptic segmentation involves a combination of joint semantic segmentation and instance segmentation, where image contents are divided into two types: things and stuff. We present Panoptic SegFormer, a general framework for panoptic segmentation with transformers. It contains three innovative components: an efficient deeply-supervised mask decoder, a query decoupling strategy, and an improved post-processing method. We also use Deformable DETR to efficiently process multi-scale features, which is a fast and efficient version of DETR. Specifically, we supervise the attention modules in the mas…
University of New Orleans（新奥尔良大学）；US Army Corps of Engineers（美国陆军工程兵团）
Imbalanced datasets are a significant challenge in real-world scenarios. They lead to models that underperform on underrepresented classes, which is a critical issue in infrastructure inspection. This paper introduces the Enhanced Feature Pyramid Network (E-FPN), a deep learning model for the semantic segmentation of culverts and sewer pipes within imbalanced datasets. The E-FPN incorporates architectural innovations like sparsely connected blocks and depth-wise separable convolutions to improve feature extraction and handle object variations. To address dataset imbalance, the model employs st…
The Hong Kong University of Science and Technology（香港科技大学）；Kuaishou Technology（快手科技）
Segmenting highly-overlapping image objects is challenging, because there is typically no distinction between real object contours and occlusion boundaries on images. Unlike previous instance segmentation methods, we model image formation as a composition of two overlapping layers, and propose Bilayer Convolutional Network (BCNet), where the top layer detects occluding objects (occluders) and the bottom layer infers partially occluded instances (occludees). The explicit modeling of occlusion relationship with bilayer structure naturally decouples the boundaries of both the occluding and occlud…
Crop-based training strategies decouple training resolution from GPU memory consumption, allowing the use of large-capacity panoptic segmentation networks on multi-megapixel images. Using crops, however, can introduce a bias towards truncating or missing large objects. To address this, we propose a novel crop-aware bounding box regression loss (CABB loss), which promotes predictions to be consistent with the visible parts of the cropped objects, while not over-penalizing them for extending outside of the crop. We further introduce a novel data sampling and augmentation strategy which improves …
DMAF-Net: An Effective Modality Rebalancing Framework for Incomplete Multi-Modal Medical Image Segmentation
College of Computer Science and Engineering, Chongqing University of Technology（重庆理工大学计算机科学与工程学院）；School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）
Semantic segmentation is the task of assigning a class-label to each pixel in an image. We propose a region-based semantic segmentation framework which handles both full and weak supervision, and addresses three common problems: (1) Objects occur at multiple scales and therefore we should use regions at multiple scales. However, these regions are overlapping which creates conflicting class predictions at the pixel-level. (2) Class frequencies are highly imbalanced in realistic datasets. (3) Each pixel can only be assigned to a single class, which creates competition between classes. We address…
King’s College London（伦敦国王学院）；University of Liverpool（利物浦大学）；Imperial College London（帝国理工学院）
Major advancements have been made in the field of object detection and segmentation recently. However, when it comes to rare categories, the state-of-the-art methods fail to detect them, resulting in a significant performance gap between rare and frequent categories. In this paper, we identify that Sigmoid or Softmax functions used in deep detectors are a major reason for low performance and are sub-optimal for long-tailed detection and segmentation. To address this, we develop a Gumbel Optimized Loss (GOL), for long-tailed detection and segmentation. It aligns with the Gumbel distribution of …
University College London (UCL)（伦敦大学学院）；King’s College London (KCL)（伦敦国王学院）；Moorfields Eye Hospital（摩尔菲尔兹眼科医院）；Institute of Ophthalmology, University College London（伦敦大学学院眼科研究所）；Wellcome/EPSRC Centre for Interventional and Surgical Sciences（惠康/英国工程与物理科学研究委员会介入与外科科学中心）；School of Biomedical Engineering & Imaging Sciences（生物医学工程与影像科学学院）
Our work proposes neural network design choices that set the state-of-the-art on a challenging public benchmark on cataract surgery, CaDIS. Our methodology achieves strong performance across three semantic segmentation tasks with increasingly granular surgical tool class sets by effectively handling class imbalance, an inherent challenge in any surgical video. We consider and evaluate two conceptually simple data oversampling methods as well as different loss functions. We show significant performance gains across network architectures and tasks especially on the rarest tool classes, thereby p…
Institute of Building Materials Science, Leibniz University Hannover（莱布尼茨汉诺威大学建筑材料科学研究所）；Institute of Photogrammetry and GeoInformation, Leibniz University Hannover（莱布尼茨汉诺威大学摄影测量与地理信息研究所）
In order to leverage and profit from unlabelled data, semi-supervised frameworks for semantic segmentation based on consistency training have been proven to be powerful tools to significantly improve the performance of purely supervised segmentation learning. However, the consensus principle behind consistency training has at least one drawback, which we identify in this paper: imbalanced label distributions within the data. To overcome the limitations of standard consistency training, we propose a novel semi-supervised framework for semantic segmentation, introducing additional losses based o…
