Deep neural networks (DNNs) for the semantic segmentation of images are usually trained to operate on a predefined closed set of object classes. This is in contrast to the "open world" setting where DNNs are envisioned to be deployed to. From a functional safety point of view, the ability to detect so-called "out-of-distribution" (OoD) samples, i.e., objects outside of a DNN's semantic space, is crucial for many applications such as automated driving. A natural baseline approach to OoD detection is to threshold on the pixel-wise softmax entropy. We present a two-step procedure that significantly improves that approach. Firstly, we utilize samples from the COCO dataset as OoD proxy and introduce a second training objective to maximize the softmax entropy on these samples. Starting from pretrained semantic segmentation networks we re-train a number of DNNs on different in-distribution datasets and consistently observe improved OoD detection performance when evaluating on completely disjoint OoD datasets. Secondly, we perform a transparent post-processing step to discard false positive OoD samples by so-called "meta classification". To this end, we apply linear models to a set of hand-crafted metrics derived from the DNN's softmax probabilities. In our experiments we consistently observe a clear additional gain in OoD detection performance, cutting down the number of detection errors by up to 52% when comparing the best baseline with our results. We achieve this improvement sacrificing only marginally in original segmentation performance. Therefore, our method contributes to safer DNNs with more reliable overall system performance.
University College London(伦敦大学学院);GSK R & D(葛兰素史克研发部)
This paper concerns pseudo labelling in segmentation. Our contribution is fourfold. Firstly, we present a new formulation of pseudo-labelling as an Expectation-Maximization (EM) algorithm for clear statistical interpretation. Secondly, we propose a semi-supervised medical image segmentation method purely based on the original pseudo labelling, namely SegPL. We demonstrate SegPL is a competitive approach against state-of-the-art consistency regularisation based methods on semi-supervised segmentation on a 2D multi-class MRI brain tumour segmentation task and a 3D binary CT lung vessel segmentat…
We propose an unsupervised superpixel segmentation method by optimizing a randomly-initialized convolutional neural network (CNN) in inference time. Our method generates superpixels via CNN from a single image without any labels by minimizing a proposed objective function for superpixel segmentation in inference time. There are three advantages to our method compared with many of existing methods: (i) leverages an image prior of CNN for superpixel segmentation, (ii) adaptively changes the number of superpixels according to the given images, and (iii) controls the property of superpixels by add…
Unsupervised pre-training has been proven as an effective approach to boost various downstream tasks given limited labeled data. Among various methods, contrastive learning learns a discriminative representation by constructing positive and negative pairs. However, it is not trivial to build reasonable pairs for a segmentation task in an unsupervised way. In this work, we propose a novel unsupervised pre-training framework that avoids the drawback of contrastive learning. Our framework consists of two principles: unsupervised over-segmentation as a pre-train task using mutual information maxim…
In this project, we study the hidden Markov random field (HMRF) model and its expectation-maximization (EM) algorithm. We implement a MATLAB toolbox named HMRF-EM-image for 2D image segmentation using the HMRF-EM framework. This toolbox also implements edge-prior-preserving image segmentation, and can be easily reconfigured for other problems, such as 3D image segmentation.
TTI-Chicago(芝加哥丰田技术研究所);Princeton University(普林斯顿大学);University of Chicago(芝加哥大学)
We study image segmentation from an information-theoretic perspective, proposing a novel adversarial method that performs unsupervised segmentation by partitioning images into maximally independent sets. More specifically, we group image pixels into foreground and background, with the goal of minimizing predictability of one set from the other. An easily computed loss drives a greedy search process to maximize inpainting error over these partitions. Our method does not involve training deep networks, is computationally cheap, class-agnostic, and even applicable in isolation to a single unlabel…
AmirKabir University of Technology(阿米尔卡比尔理工大学);Sharif University of Technology(谢里夫理工大学);Monash University(莫纳什大学)
Semantic segmentation is one of the basic, yet essential scene understanding tasks for an autonomous agent. The recent developments in supervised machine learning and neural networks have enjoyed great success in enhancing the performance of the state-of-the-art techniques for this task. However, their superior performance is highly reliant on the availability of a large-scale annotated dataset. In this paper, we propose a novel fully unsupervised semantic segmentation method, the so-called Information Maximization and Adversarial Regularization Segmentation (InMARS). Inspired by human percept…
Kidney volume is greatly affected in several renal diseases. Precise and automatic segmentation of the kidney can help determine kidney size and evaluate renal function. Fully convolutional neural networks have been used to segment organs from large biomedical 3D images. While these networks demonstrate state-of-the-art segmentation performances, they do not immediately translate to small foreground objects, small sample sizes, and anisotropic resolution in MRI datasets. In this paper we propose a new framework to address some of the challenges for segmenting 3D MRI. These methods were impleme…
University of Pennsylvania(宾夕法尼亚大学);Stanford University(斯坦福大学)
We introduce Equivariant Neural Field Expectation Maximization (EFEM), a simple, effective, and robust geometric algorithm that can segment objects in 3D scenes without annotations or training on scenes. We achieve such unsupervised segmentation by exploiting single object shape priors. We make two novel steps in that direction. First, we introduce equivariant shape representations to this problem to eliminate the complexity induced by the variation in object configuration. Second, we propose a novel EM algorithm that can iteratively refine segmentation masks using the equivariant shape prior.…
Institute of Computer Graphics and Vision, Graz University of Technology(格拉茨技术大学计算机图形学与视觉研究所);Graz University of Technology(格拉茨技术大学)
We propose a novel method for unsupervised semantic image segmentation based on mutual information maximization between local and global high-level image features. The core idea of our work is to leverage recent progress in self-supervised image representation learning. Representation learning methods compute a single high-level feature capturing an entire image. In contrast, we compute multiple high-level features, each capturing image segments of one particular semantic class. To this end, we propose a novel two-step learning procedure comprising a segmentation and a mutual information maxim…
ZScribbleSeg: A comprehensive segmentation framework with modeling of efficient annotation and maximization of scribble supervision
School of Data Science, Fudan University, Shanghai, 200433, China(复旦大学数据科学学院);Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, USA(约翰霍普金斯大学电气与计算机工程系)
Tencent AI Lab(腾讯人工智能实验室);Tencent Data Platform(腾讯数据平台部);Tsinghua University(清华大学);Tsinghua Shenzhen International Graduate School(清华大学深圳国际研究生院);Peng Cheng Lab(鹏城实验室)
Matching-based methods, especially those based on space-time memory, are significantly ahead of other solutions in semi-supervised video object segmentation (VOS). However, continuously growing and redundant template features lead to an inefficient inference. To alleviate this, we propose a novel Sequential Weighted Expectation-Maximization (SWEM) network to greatly reduce the redundancy of memory features. Different from the previous methods which only detect feature redundancy between frames, SWEM merges both intra-frame and inter-frame similar features by leveraging the sequential weighted …
Iterative Event-based Motion Segmentation by Variational Contrast Maximization
Keio University(庆应大学);Technische Universität Berlin(柏林技术大学);Einstein Center Digital Future(爱因斯坦数字未来研究中心);Science of Intelligence Excellence Cluster, Germany(德国智能科学卓越集群)
Uncertainty Estimation for Safety-critical Scene Segmentation via Fine-grained Reward Maximization
The Chinese University of Hong Kong(香港中文大学);University Hospital of Augsburg(奥格斯堡大学医院);Harvard Medical School & Massachusetts General Hospital(哈佛医学院与麻省总医院)
University of Cambridge(剑桥大学);Shanghai General Hospital(上海市第一人民医院);Shanghai Jiao Tong University(上海交通大学);Southern University of Science and Technology(南方科技大学);University of Bath(巴斯大学)
We present an Expectation-Maximization (EM) Regularized Deep Learning (EMReDL) model for weakly supervised tumor segmentation. The proposed framework is tailored to glioblastoma, a type of malignant tumor characterized by its diffuse infiltration into the surrounding brain tissue, which poses significant challenge to treatment target and tumor burden estimation using conventional structural MRI. Although physiological MRI provides more specific information regarding tumor infiltration, the relatively low resolution hinders a precise full annotation. This has motivated us to develop a weakly su…
ÉTS Montréal(蒙特利尔高等技术学院);Centre de recherche du Centre hospitalier de l’Université de Montréal (CRCHUM)(蒙特利尔大学医院研究中心(CRCHUM))
Despite achieving promising results in a breadth of medical image segmentation tasks, deep neural networks require large training datasets with pixel-wise annotations. Obtaining these curated datasets is a cumbersome process which limits the applicability in scenarios. Mixed supervision is an appealing alternative for mitigating this obstacle. In this work, we propose a dual-branch architecture, where the upper branch (teacher) receives strong annotations, while the bottom one (student) is driven by limited supervision and guided by the upper branch. Combined with a standard cross-entropy loss…
Computer Vision Center(计算机视觉中心);Universitat Autònoma de Barcelona(巴塞罗那自治大学)
We successfully combine Expectation-Maximization algorithm and variational approaches for parameter learning and computing inference on Markov random felds. This is a general method that can be applied to many computer vision tasks. In this paper, we apply it to handwritten text line segmentation. We conduct several experiments that demonstrate that our method deal with common issues of this task, such as complex document layout or non-latin scripts. The obtained results prove that our method achieve state-of-the-art performance on different benchmark datasets without any particular fine tunin…
Meply: A Large-scale Dataset and Baseline Evaluations for Metastatic Perirectal Lymph Node Detection and Segmentation
School of Computer Science and Technology, University of Science and Technology of China(中国科学技术大学计算机科学与技术学院);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院);The First Affiliated Hospital of Anhui Medical University(安徽医科大学第一附属医院);Anhui Medical University(安徽医科大学)
Universidade Federal de Minas Gerais(米纳斯吉拉斯联邦大学);University of São Paulo(圣保罗大学);Federal University of Mato Grosso do Sul(南马托格罗索联邦大学)
Semantic segmentation is a classic computer vision task with multiple applications, which includes medical and remote sensing image analysis. Despite recent advances with deep-based approaches, labeling samples (pixels) for training models is laborious and, in some cases, unfeasible. In this paper, we present two novel meta learning methods, named WeaSeL and ProtoSeg, for the few-shot semantic segmentation task with sparse annotations. We conducted extensive evaluation of the proposed methods in different applications (12 datasets) in medical imaging and agricultural remote sensing, which are …
Birla Institute of Technology and Science, Pilani(皮拉尼比尔拉技术与科学学院)
Modern deep learning models have revolutionized the field of computer vision. But, a significant drawback of most of these models is that they require a large number of labelled examples to generalize properly. Recent developments in few-shot learning aim to alleviate this requirement. In this paper, we propose a novel lightweight CNN architecture for 1-shot image segmentation. The proposed model is created by taking inspiration from well-performing architectures for semantic segmentation and adapting it to the 1-shot domain. We train our model using 4 meta-learning algorithms that have worked…
Metadata Improves Segmentation Through Multitasking Elicitation
Ukrainian Catholic University(乌克兰天主教大学);Leibniz Institute of Plant Genetics and Crop Plant Research (IPK)(莱布尼茨植物遗传与作物研究所(IPK));University of Tartu(塔尔图大学);Revvity, Inc.(Revvity公司);Wellcome Sanger Institute(威康桑格研究所)
Inria Grand Est(法国国家信息与自动化研究所大东部分部);Université de Lorraine(洛林大学)
This paper presents an efficient approach for solving jointly facade registration and semantic segmentation. Progress in facade detection and recognition enable good initialization for the registration of a reference facade to a newly acquired target image. We propose here to rely on semantic segmentation to improve the accuracy of that initial registration. Simultaneously we aim to improve the quality of the semantic segmentation through the registration. These two problems are jointly solved in a Expectation-Maximization framework. We especially introduce a bayesian model that use prior sema…
Nanjing University(南京大学);University of Wollongong(伍伦贡大学)
To address the annotation scarcity issue in some cases of semantic segmentation, there have been a few attempts to develop the segmentation model in the few-shot learning paradigm. However, most existing methods only focus on the traditional 1-way segmentation setting (i.e., one image only contains a single object). This is far away from practical semantic segmentation tasks where the K-way setting (K>1) is usually required by performing the accurate multi-object segmentation. To deal with this issue, we formulate the few-shot semantic segmentation task as a learning-based pixel classification…
NYU Multimedia and Visual Computing Lab(纽约大学多媒体与视觉计算实验室);New York University Abu Dhabi(纽约大学阿布扎比分校);New York University(纽约大学)
Though deep learning methods have shown great success in 3D point cloud part segmentation, they generally rely on a large volume of labeled training data, which makes the model suffer from unsatisfied generalization abilities to unseen classes with limited data. To address this problem, we present a novel meta-learning strategy that regards the 3D shape segmentation function as a task. By training over a number of 3D part segmentation tasks, our method is capable to learn the prior over the respective 3D segmentation function space which leads to an optimal model that is rapidly adapting to ne…
Technical University of Munich(慕尼黑工业大学);Ulm University(乌尔姆大学)
The lack of sufficient annotated image data is a common issue in medical image segmentation. For some organs and densities, the annotation may be scarce, leading to poor model training convergence, while other organs have plenty of annotated data. In this work, we present MetaMedSeg, a gradient-based meta-learning algorithm that redefines the meta-learning task for the volumetric medical data with the goal to capture the variety between the slices. We also explore different weighting schemes for gradients aggregation, arguing that different tasks might have different complexity, and hence, con…
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学)
Supervised training of deep neural networks (DNNs) by noisy labels has been studied extensively in image classification but much less in image segmentation. Our understanding of the learning behavior of DNNs trained by noisy segmentation labels remains limited. We address this deficiency in both binary segmentation of biological microscopy images and multi-class segmentation of natural images. We classify segmentation labels according to their noise transition matrices (NTMs) and compare performance of DNNs trained by different types of labels. When we randomly sample a small fraction (e.g., 1…
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学)
How deep neural networks (DNNs) learn from noisy labels has been studied extensively in image classification but much less in image segmentation. So far, our understanding of the learning behavior of DNNs trained by noisy segmentation labels remains limited. In this study, we address this deficiency in both binary segmentation of biological microscopy images and multi-class segmentation of natural images. We generate extremely noisy labels by randomly sampling a small fraction (e.g., 10%) or flipping a large fraction (e.g., 90%) of the ground truth labels. When trained with these noisy labels,…
Deep convolutional neural networks (DCNNs) trained on a large number of images with strong pixel-level annotations have recently significantly pushed the state-of-art in semantic image segmentation. We study the more challenging problem of learning DCNNs for semantic image segmentation from either (1) weakly annotated training data such as bounding boxes or image-level labels or (2) a combination of few strongly labeled and many weakly labeled images, sourced from one or multiple datasets. We develop Expectation-Maximization (EM) methods for semantic image segmentation model training under the…
Deep neural networks have achieved satisfactory performance in piles of medical image analysis tasks. However the training of deep neural network requires a large amount of samples with high-quality annotations. In medical image segmentation, it is very laborious and expensive to acquire precise pixel-level annotations. Aiming at training deep segmentation models on datasets with probably corrupted annotations, we propose a novel Meta Corrupted Pixels Mining (MCPM) method based on a simple meta mask network. Our method is targeted at automatically estimate a weighting map to evaluate the impor…
We extend first-order model agnostic meta-learning algorithms (including FOMAML and Reptile) to image segmentation, present a novel neural network architecture built for fast learning which we call EfficientLab, and leverage a formal definition of the test error of meta-learning algorithms to decrease error on out of distribution tasks. We show state of the art results on the FSS-1000 dataset by meta-training EfficientLab with FOMAML and using Bayesian optimization to infer the optimal test-time adaptation routine hyperparameters. We also construct a small benchmark dataset, FP-k, for the empi…
National Taiwan University(台湾大学);ASUS Intelligent Cloud Services(华硕智能云服务)
How to handle domain shifts when recognizing or segmenting visual data across domains has been studied by learning and vision communities. In this paper, we address domain generalized semantic segmentation, in which the segmentation model is trained on multiple source domains and is expected to generalize to unseen data domains. We propose a novel meta-learning scheme with feature disentanglement ability, which derives domain-invariant features for semantic segmentation with domain generalization guarantees. In particular, we introduce a class-specific feature critic module in our framework, e…
Unsupervised image segmentation is an important task in many real-world scenarios where labelled data is of scarce availability. In this paper we propose a novel approach that harnesses recent advances in unsupervised learning using a combination of Mutual Information Maximization (MIM), Neural Superpixel Segmentation and Graph Neural Networks (GNNs) in an end-to-end manner, an approach that has not been explored yet. We take advantage of the compact representation of superpixels and combine it with GNNs in order to learn strong and semantically meaningful representations of images. Specifical…
National Taiwan University(台湾大学);ASUS Intelligent Cloud Services(华硕智能云服务)
Few-shot semantic segmentation addresses the learning task in which only few images with ground truth pixel-level labels are available for the novel classes of interest. One is typically required to collect a large mount of data (i.e., base classes) with such ground truth information, followed by meta-learning strategies to address the above learning task. When only image-level semantic labels can be observed during both training and testing, it is considered as an even more challenging task of weakly supervised few-shot semantic segmentation. To address this problem, we propose a novel meta-l…
University of Chinese Academy of Sciences(中国科学院大学)
Few-shot segmentation is challenging because objects within the support and query images could significantly differ in appearance and pose. Using a single prototype acquired directly from the support image to segment the query image causes semantic ambiguity. In this paper, we propose prototype mixture models (PMMs), which correlate diverse image regions with multiple prototypes to enforce the prototype-based semantic representation. Estimated by an Expectation-Maximization algorithm, PMMs incorporate rich channel-wised and spatial semantics from limited support images. Utilized as representat…
Fujian Key Laboratory of Sensing and Computing for Smart City, School of Informatics(智慧城市感知与计算福建省重点实验室、信息学院);School of Computer Science and Technology, East China Normal University(华东师范大学计算机科学与技术学院)
Despite recent progress on semantic segmentation, there still exist huge challenges in medical ultra-resolution image segmentation. The methods based on multi-branch structure can make a good balance between computational burdens and segmentation accuracy. However, the fusion structure in these methods require to be designed elaborately to achieve desirable result, which leads to model redundancy. In this paper, we propose Meta Segmentation Network (MSN) to solve this challenging problem. With the help of meta-learning, the fusion module of MSN is quite simple but effective. MSN can fast gener…
National Technical University of Athens(雅典国家技术大学);University of West Attica(西阿提卡大学)
Corrosion detection on metal constructions is a major challenge in civil engineering for quick, safe and effective inspection. Existing image analysis approaches tend to place bounding boxes around the defected region which is not adequate both for structural analysis and pre-fabrication, an innovative construction concept which reduces maintenance cost, time and improves safety. In this paper, we apply three semantic segmentation-oriented deep learning models (FCN, U-Net and Mask R-CNN) for corrosion detection, which perform better in terms of accuracy and time and require a smaller number of…
School of Engineering, Nazarbayev University(纳扎尔巴耶夫大学工程学院)
The automated segmentation of cells in microscopic images is an open research problem that has important implications for studies of the developmental and cancer processes based on in vitro models. In this paper, we present the approach for segmentation of the DIC images of cultured cells using G-neighbor smoothing followed by Kauwahara filtering and local standard deviation approach for boundary detection. NIH FIJI/ImageJ tools are used to create the ground truth dataset. The results of this work indicate that detection of cell boundaries using segmentation approach even in the case of realis…
S-lab, Nanyang Technological University(南洋理工大学S-lab);School of Computer Science and Engineering, Nanyang Technological University(南洋理工大学计算机科学与工程学院);Electrical Engineering and Computer Science, Oregon State University(俄勒冈州立大学电气工程与计算机科学学院);Dept of Data Science and AI, Monash University(莫纳什大学数据科学与人工智能系)
Currently, the state-of-the-art methods treat few-shot semantic segmentation task as a conditional foreground-background segmentation problem, assuming each class is independent. In this paper, we introduce the concept of meta-class, which is the meta information (e.g. certain middle-level features) shareable among all classes. To explicitly learn meta-class representations in few-shot segmentation task, we propose a novel Meta-class Memory based few-shot segmentation method (MM-Net), where we introduce a set of learnable memory embeddings to memorize the meta-class information during the base…
The Chinese University of Hong Kong(香港中文大学);Chinese Academy of Sciences(中国科学院)
Model generalization capacity at domain shift (e.g., various imaging protocols and scanners) is crucial for deep learning methods in real-world clinical deployment. This paper tackles the challenging problem of domain generalization, i.e., learning a model from multi-domain source data such that it can directly generalize to an unseen target domain. We present a novel shape-aware meta-learning scheme to improve the model generalization in prostate MRI segmentation. Our learning scheme roots in the gradient-based meta-learning, by explicitly simulating domain shift with virtual meta-train and m…
Zhejiang University(浙江大学);University of Technology Sydney(悉尼科技大学);Baidu Research(百度研究院)
Prevalent semantic segmentation solutions are, in essence, a dense discriminative classifier of p(class|pixel feature). Though straightforward, this de facto paradigm neglects the underlying data distribution p(pixel feature|class), and struggles to identify out-of-distribution data. Going beyond this, we propose GMMSeg, a new family of segmentation models that rely on a dense generative classifier for the joint distribution p(pixel feature,class). For each class, GMMSeg builds Gaussian Mixture Models (GMMs) via Expectation-Maximization (EM), so as to capture class-conditional densities. Meanw…
Few-shot learning is a standard practice in most deep learning based histopathology image segmentation, given the relatively low number of digitized slides that are generally available. While many models have been developed for domain specific histopathology image segmentation, cross-domain generalization remains a key challenge for properly validating models. Here, tooling and datasets to benchmark model performance across histopathological domains are lacking. To address this limitation, we introduce MetaHistoSeg - a Python framework that implements unique scenarios in both meta learning and…
Xi’an Jiao tong University(西安交通大学);The First Affiliated Hospital of Xi’an Jiaotong University(西安交通大学第一附属医院)
Nuclei segmentation is a fundamental task in digital pathology analysis and can be automated by deep learning-based methods. However, the development of such an automated method requires a large amount of data with precisely annotated masks which is hard to obtain. Training with weakly labeled data is a popular solution for reducing the workload of annotation. In this paper, we propose a novel meta-learning-based nuclei segmentation method which follows the label correction paradigm to leverage data with noisy masks. Specifically, we design a fully conventional meta-model that can correct nois…
MonolithAI;Middle East Technical University(中东技术大学);Hacettepe University(哈切特佩大学)
Just like other few-shot learning problems, few-shot segmentation aims to minimize the need for manual annotation, which is particularly costly in segmentation tasks. Even though the few-shot setting reduces this cost for novel test classes, there is still a need to annotate the training data. To alleviate this need, we propose a self-supervised training approach for learning few-shot segmentation models. We first use unsupervised saliency estimation to obtain pseudo-masks on images. We then train a simple prototype based model over different splits of pseudo masks and augmentations of images.…
The significant amount of training data required for training Convolutional Neural Networks has become a bottleneck for applications like semantic segmentation. Few-shot semantic segmentation algorithms address this problem, with an aim to achieve good performance in the low-data regime, with few annotated training images. Recently, approaches based on class-prototypes computed from available training data have achieved immense success for this task. In this work, we propose a novel meta-learning framework, Semantic Meta-Learning (SML) which incorporates class level semantic descriptions in th…
Institute of Information Science, Academia Sinica(中央研究院信息科学研究所);National Tsing Hua University(国立清华大学)
This paper presents a "learning to learn" approach to figure-ground image segmentation. By exploring webly-abundant images of specific visual effects, our method can effectively learn the visual-effect internal representations in an unsupervised manner and uses this knowledge to differentiate the figure from the ground in an image. Specifically, we formulate the meta-learning process as a compositional image editing task that learns to imitate a certain visual effect and derive the corresponding internal representation. Such a generative process can help instantiate the underlying figure-groun…
University of Edinburgh(爱丁堡大学);Canon Medical Research Europe Ltd.(佳能医疗欧洲研究有限公司);The Alan Turing Institute(艾伦·图灵研究所)
Generalising deep models to new data from new centres (termed here domains) remains a challenge. This is largely attributed to shifts in data statistics (domain shifts) between source and unseen domains. Recently, gradient-based meta-learning approaches where the training data are split into meta-train and meta-test sets to simulate and handle the domain shifts during training have shown improved generalisation performance. However, the current fully supervised meta-learning approaches are not scalable for medical image segmentation, where large effort is required to create pixel-wise annotati…