FU-net: Multi-class Image Segmentation Using Feedback Weighted U-net
School of Computer Science, University of Nottingham(诺丁汉大学计算机学院);School of Medicine, University of Nottingham(诺丁汉大学医学院);Sir Peter Mansfield Imaging Centre, University of Nottingham(诺丁汉大学彼得·曼斯菲尔德爵士成像中心)
Mina Jafari, Ruizhe Li, Yue Xing, Dorothee Auer, Susan Francis, Jonathan Garibaldi, Xin Chen
备注:
Accepted for publication at International Conference on Image and Graphics (ICIG 2019)
发表信息:
The 10th International Conference on Image and Graphics (ICIG 2019)
In this paper, we present a generic deep convolutional neural network (DCNN) for multi-class image segmentation. It is based on a well-established supervised end-to-end DCNN model, known as U-net. U-net is firstly modified by adding widely used batch normalization and residual block (named as BRU-net) to improve the efficiency of model training. Based on BRU-net, we further introduce a dynamically weighted cross-entropy loss function. The weighting scheme is calculated based on the pixel-wise prediction accuracy during the training process. Assigning higher weights to pixels with lower segmentation accuracies enables the network to learn more from poorly predicted image regions. Our method is named as feedback weighted U-net (FU-net). We have evaluated our method based on T1- weighted brain MRI for the segmentation of midbrain and substantia nigra, where the number of pixels in each class is extremely unbalanced to each other. Based on the dice coefficient measurement, our proposed FU-net has outperformed BRU-net and U-net with statistical significance, especially when only a small number of training examples are available. The code is publicly available in GitHub (GitHub link: https://github.com/MinaJf/FU-net).
We present Spectral GUI, a multiplatform breast MR image analysis tool designed to facilitate the segmentation of fibro glandular tissues and lesions in T1 weighted breast MR images via a graphical user interface (GUI). Spectral GUIR uses spectrum loft method [1] for breast MR image segmentation. Not only is it interactive, but robust and expeditious at the same time. Being devoid of any machine learning algorithm, it shows exceptionally high execution speed with minimal overheads. The accuracy of the results has been simultaneously measured using performance metrics and expert entailment. The…
Partial Decoder Attention Network with Contour-weighted Loss Function for Data-Imbalance Medical Image Segmentation
Institute of Medical Technology, Peking University Health Science Center, Peking University, Beijing, China(北京大学医学部医学技术研究所);National Institute of Health Data Science, Peking University, Beijing, China(北京大学国家健康数据科学研究院);Department of Radiology, Peking University Third Hospital, Beijing, China(北京大学第三医院放射科);RIKEN Center for Computational Science (R-CCS), Kobe, Japan(日本京都大学RIKEN计算科学中心);National Institute of Health Data Science, Peking University, the Institute of Medical Technology, Peking University Health Science Center, and the Institute for Artificial Intelligence, Peking University, Beijing, China(北京大学国家健康数据科学研究院、北京大学医学部医学技术研究所以及北京大学人工智能研究所)
Max Planck Institute of Molecular Cell Biology and Genetics(马克斯·普朗克分子细胞生物学与遗传学研究所);Center for Systems Biology Dresden(德累斯顿系统生物学中心)
Convolutional neural networks have become state-of-the-art in a wide range of image recognition tasks. The interpretation of their predictions, however, is an active area of research. Whereas various interpretation methods have been suggested for image classification, the interpretation of image segmentation still remains largely unexplored. To that end, we propose SEG-GRAD-CAM, a gradient-based method for interpreting semantic segmentation. Our method is an extension of the widely-used Grad-CAM method, applied locally to produce heatmaps showing the relevance of individual pixels for semantic…
Accurate segmentation of the prostate from magnetic resonance (MR) images provides useful information for prostate cancer diagnosis and treatment. However, automated prostate segmentation from 3D MR images still faces several challenges. For instance, a lack of clear edge between the prostate and other anatomical structures makes it challenging to accurately extract the boundaries. The complex background texture and large variation in size, shape and intensity distribution of the prostate itself make segmentation even further complicated. With deep learning, especially convolutional neural net…
The recent development of light-weighted neural networks has promoted the applications of deep learning under resource constraints and mobile applications. Many of these applications need to perform a real-time and efficient prediction for semantic segmentation with a light-weighted network. This paper introduces a light-weighted network with an efficient reduced non-local module (LRNNet) for efficient and realtime semantic segmentation. We proposed a factorized convolutional block in ResNet-Style encoder to achieve more lightweighted, efficient and powerful feature extraction. Meanwhile, our …
University of California at Irvine(加州大学欧文分校);Whittier College(惠蒂尔学院);University of Texas at Dallas(德克萨斯大学达拉斯分校)
In a class of piecewise-constant image segmentation models, we propose to incorporate a weighted difference of anisotropic and isotropic total variation (AITV) to regularize the partition boundaries in an image. In particular, we replace the total variation regularization in the Chan-Vese segmentation model and a fuzzy region competition model by the proposed AITV. To deal with the nonconvex nature of AITV, we apply the difference-of-convex algorithm (DCA), in which the subproblems can be minimized by the primal-dual hybrid gradient method with linesearch. The convergence of the DCA scheme is …
Minimization of boundary curvature is a classic regularization technique for image segmentation in the presence of noisy image data. Techniques for minimizing curvature have historically been derived from descent methods which could be trapped in a local minimum and therefore required a good initialization. Recently, combinatorial optimization techniques have been applied to the optimization of curvature which provide a solution that achieves nearly a global optimum. However, when applied to image segmentation these methods required a meaningful data term. Unfortunately, for many images, parti…
Purpose: The purpose of this study is to investigate the robustness of a commonly-used convolutional neural network for image segmentation with respect to visually-subtle adversarial perturbations, and suggest new methods to make these networks more robust to such perturbations. Materials and Methods: In this retrospective study, the accuracy of brain tumor segmentation was studied in subjects with low- and high-grade gliomas. A three-dimensional UNet model was implemented to segment four different MR series (T1-weighted, post-contrast T1-weighted, T2- weighted, and T2-weighted FLAIR) into fou…
Image segmentation in the medical domain is a challenging field owing to poor resolution and limited contrast. The predominantly used conventional segmentation techniques and the thresholding methods suffer from limitations because of heavy dependence on user interactions. Uncertainties prevalent in an image cannot be captured by these techniques. The performance further deteriorates when the images are corrupted by noise, outliers and other artifacts. The objective of this paper is to develop an effective robust fuzzy C- means clustering for segmenting vertebral body from magnetic resonance i…
SPINEPS -- Automatic Whole Spine Segmentation of T2-weighted MR images using a Two-Phase Approach to Multi-class Semantic and Instance Segmentation
Technical University of Munich(慕尼黑工业大学);Klinikum rechts der Isar(伊萨尔河右岸医院);University of Zurich(苏黎世大学);Imperial College London(帝国理工学院);Helmholtz Munich(亥姆霍兹慕尼黑中心);University Medical Center Rostock(罗斯托克大学医学中心);University of Freiburg(弗赖堡大学);University Hospital Augsburg(奥格斯堡大学医院);Augsburg University(奥格斯堡大学)
Beckman Institute at Caltech(加州理工学院贝克曼研究所);Center for Advanced Methods in Biological Image Analysis(生物图像分析高级方法中心)
We propose a new multiclass weighted loss function for instance segmentation of cluttered cells. We are primarily motivated by the need of developmental biologists to quantify and model the behavior of blood T-cells which might help us in understanding their regulation mechanisms and ultimately help researchers in their quest for developing an effective immuno-therapy cancer treatment. Segmenting individual touching cells in cluttered regions is challenging as the feature distribution on shared borders and cell foreground are similar thus difficulting discriminating pixels into proper classes.…
We propose an automatic preprocessing and ensemble learning for segmentation of cell images with low quality. It is difficult to capture cells with strong light. Therefore, the microscopic images of cells tend to have low image quality but these images are not good for semantic segmentation. Here we propose a method to translate an input image to the images that are easy to recognize by deep learning. The proposed method consists of two deep neural networks. The first network is the usual training for semantic segmentation, and penultimate feature maps of the first network are used as filters …
Research into Few-shot Semantic Segmentation (FSS) has attracted great attention, with the goal to segment target objects in a query image given only a few annotated support images of the target class. A key to this challenging task is to fully utilize the information in the support images by exploiting fine-grained correlations between the query and support images. However, most existing approaches either compressed the support information into a few class-wise prototypes, or used partial support information (e.g., only foreground) at the pixel level, causing non-negligible information loss. …
Australian National University(澳大利亚国立大学);SenseTime(商汤科技)
Weakly Supervised Semantic Segmentation (WSSS) is challenging, particularly when image-level labels are used to supervise pixel level prediction. To bridge their gap, a Class Activation Map (CAM) is usually generated to provide pixel level pseudo labels. CAMs in Convolutional Neural Networks suffer from partial activation ie, only the most discriminative regions are activated. Transformer based methods, on the other hand, are highly effective at exploring global context with long range dependency modeling, potentially alleviating the "partial activation" issue. In this paper, we propose the fi…
Max-Delbrueck-Center for Molecular Medicine in the Helmholtz Association (MDC)(亥姆霍兹联合会马克斯·德尔布吕克分子医学中心);Humboldt-Universität zu Berlin(柏林洪堡大学);Case Western Reserve University(凯斯西储大学);Lausanne University Hospital(洛桑大学医院);University of Bern(伯尔尼大学);Charité University Medicine(夏里特医科大学)
We describe here the panoptic segmentation method we devised for our participation in the CoNIC: Colon Nuclei Identification and Counting Challenge at ISBI 2022. Key features of our method are a weighted loss specifically engineered for semantic segmentation of highly imbalanced cell types, and a state-of-the art nuclei instance segmentation model, which we combine in a Hovernet-like architecture.
Despite the popularity and empirical success of patch-based nearest-neighbor and weighted majority voting approaches to medical image segmentation, there has been no theoretical development on when, why, and how well these nonparametric methods work. We bridge this gap by providing a theoretical performance guarantee for nearest-neighbor and weighted majority voting segmentation under a new probabilistic model for patch-based image segmentation. Our analysis relies on a new local property for how similar nearby patches are, and fuses existing lines of work on modeling natural imagery patches a…
The paper proposes a novel Kernelized image segmentation scheme for noisy images that utilizes the concept of Smallest Univalue Segment Assimilating Nucleus (SUSAN) and incorporates spatial constraints by computing circular colour map induced weights. Fuzzy damping coefficients are obtained for each nucleus or center pixel on the basis of the corresponding weighted SUSAN area values, the weights being equal to the inverse of the number of horizontal and vertical moves required to reach a neighborhood pixel from the center pixel. These weights are used to vary the contributions of the different…
Location information is proven to benefit the deep learning models on capturing the manifold structure of target objects, and accordingly boosts the accuracy of medical image segmentation. However, most existing methods encode the location information in an implicit way, e.g. the distance transform maps, which describe the relative distance from each pixel to the contour boundary, for the network to learn. These implicit approaches do not fully exploit the position information (i.e. absolute location) of targets. In this paper, we propose a novel loss function, namely residual moment (RM) loss…
Enhancing sea ice segmentation in Sentinel-1 images with atrous convolutions
University of Colorado Boulder(科罗拉多大学博尔德分校);Norwegian Meteorological Institute(挪威气象研究所);National Snow and Ice Data Center(国家冰雪数据中心);CIRES(CIRES(环境科学研究合作研究所))
本文针对 Sentinel-1 SAR 海冰制图自动化难题,提出结合 ResNet 与空洞空间金字塔池化的分割方法,在 Extreme Earth version 2 数据集上实现高效、抗噪的海冰与冰型分类,性能优于基线 U-Net。
University of Texas at Austin(德克萨斯大学奥斯汀分校);Snap Inc.(Snap公司)
Concept shift is a prevailing problem in natural tasks like medical image segmentation where samples usually come from different subpopulations with variant correlations between features and labels. One common type of concept shift in medical image segmentation is the "information imbalance" between label-sparse samples with few (if any) segmentation labels and label-dense samples with plentiful labeled pixels. Existing distributionally robust algorithms have focused on adaptively truncating/down-weighting the "less informative" (i.e., label-sparse in our context) samples. To exploit data feat…
Object segmentation plays an important role in the modern medical image analysis, which benefits clinical study, disease diagnosis, and surgery planning. Given the various modalities of medical images, the automated or semi-automated segmentation approaches have been used to identify and parse organs, bones, tumors, and other regions-of-interest (ROI). However, these contemporary segmentation approaches tend to fail to predict the boundary areas of ROI, because of the fuzzy appearance contrast caused during the imaging procedure. To further improve the segmentation quality of boundary areas, w…
Low-Rank Continual Pyramid Vision Transformer: Incrementally Segment Whole-Body Organs in CT with Light-Weighted Adaptation
DAMO Academy, Alibaba Group(阿里巴巴达摩院);State University of New York at Stony Brook(纽约州立大学石溪分校);Johns Hopkins University(约翰斯·霍普金斯大学);The First Affiliated Hospital Zhejiang University(浙江大学医学院附属第一医院)
Department of Electrical Engineering (ESAT), Center for Processing Speech and Images, KU Leuven(比利时鲁汶大学(KU Leuven)电气工程系(ESAT)、语音与图像处理中心);Department of Gastroenterology and Hepatology, UZ Leuven(比利时鲁汶大学UZ(UZ Leuven)消化内科和肝病科)
In many medical imaging and classical computer vision tasks, the Dice score and Jaccard index are used to evaluate the segmentation performance. Despite the existence and great empirical success of metric-sensitive losses, i.e. relaxations of these metrics such as soft Dice, soft Jaccard and Lovasz-Softmax, many researchers still use per-pixel losses, such as (weighted) cross-entropy to train CNNs for segmentation. Therefore, the target metric is in many cases not directly optimized. We investigate from a theoretical perspective, the relation within the group of metric-sensitive loss functions…
CT perfusion (CTP) has been used to triage ischemic stroke patients in the early stage, because of its speed, availability, and lack of contraindications. Perfusion parameters including cerebral blood volume (CBV), cerebral blood flow (CBF), mean transit time (MTT) and time of peak (Tmax) could also be computed from CTP data. However, CTP data or the perfusion parameters, are ambiguous to locate the infarct core or tissue at risk (penumbra), which is normally confirmed by the follow-up Diffusion Weighted Imaging (DWI) or perfusion diffusion mismatch. In this paper, we propose a novel generativ…
The aim of this study is to investigate the segmentation accuracies of different segmentation networks trained on 730 manually annotated lateral lumbar spine X-rays. Instance segmentation networks were compared to semantic segmentation networks. The study cohort comprised diseased spines and postoperative images with metallic implants. The average mean accuracy and mean intersection over union (IoU) was up to 3 percent better for the best performing instance segmentation model, the average pixel accuracy and weighted IoU were slightly better for the best performing semantic segmentation model.…
University for Health Sciences, Medical Informatics and Technology (UMIT)(健康科学、医学信息与技术大学(UMIT))
We study the applicability of a set of texture descriptors introduced in recent work by the author to texture-based segmentation of images. The texture descriptors under investigation result from applying graph indices from quantitative graph theory to graphs encoding the local structure of images. The underlying graphs arise from the computation of morphological amoebas as structuring elements for adaptive morphology, either as weighted or unweighted Dijkstra search trees or as edge-weighted pixel graphs within structuring elements. In the present paper we focus on texture descriptors in whic…
Automated Olfactory Bulb Segmentation on High Resolutional T2-Weighted MRI
German Center for Neurodegenerative Diseases (DZNE)(德国神经退行性疾病研究中心);University of Bonn(波恩大学);Institute for Medical Biometry, Informatics and Epidemiology (IMBIE)(医学生物统计学、信息学与流行病学研究所);A.A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital(麻省总医院A.A.马蒂诺生物医学成像中心);Harvard Medical School(哈佛医学院)
The Dice score and Jaccard index are commonly used metrics for the evaluation of segmentation tasks in medical imaging. Convolutional neural networks trained for image segmentation tasks are usually optimized for (weighted) cross-entropy. This introduces an adverse discrepancy between the learning optimization objective (the loss) and the end target metric. Recent works in computer vision have proposed soft surrogates to alleviate this discrepancy and directly optimize the desired metric, either through relaxations (soft-Dice, soft-Jaccard) or submodular optimization (Lovász-softmax). The aim …
University of Nottingham(诺丁汉大学);University of Leicester(莱斯特大学)
Optical coherence tomography (OCT) is a non-invasive imaging technique that can produce images of the eye at the microscopic level. OCT image segmentation to localise retinal layer boundaries is a fundamental procedure for diagnosing and monitoring the progression of retinal and optical nerve disorders. In this paper, we introduce a novel and accurate geodesic distance method (GDM) for OCT segmentation of both healthy and pathological images in either two- or three-dimensional spaces. The method uses a weighted geodesic distance by an exponential function, taking into account both horizontal a…
Disentangling Sampling from Training Budget in Class-Imbalanced CT Body Composition Segmentation
Amsterdam University Medical Center(阿姆斯特丹大学医学中心);University of Amsterdam(阿姆斯特丹大学);Informatics Institute, Faculty of Science(信息学院,科学学院);Department of Radiology(放射科);Mayo Clinic Rochester(罗切斯特梅奥诊所)
SWCF-Net: Similarity-weighted Convolution and Local-global Fusion for Efficient Large-scale Point Cloud Semantic Segmentation
Biomimetic and Intelligent Robotics Lab (BIRL), School of Electromechanical Engineering, Guangdong University of Technology(广东工业大学机电工程学院仿生与智能机器人实验室(BIRL));Southern University of Science and Technology(南方科技大学);Meituan Technology Co., Ltd(美团科技有限公司)
We focus on the very challenging task of semantic segmentation for autonomous driving system. It must deliver decent semantic segmentation result for traffic critical objects real-time. In this paper, we propose a very efficient yet powerful deep neural network for driving scene semantic segmentation termed as Driving Segmentation Network (DSNet). DSNet achieves state-of-the-art balance between accuracy and inference speed through efficient units and architecture design inspired by ShuffleNet V2 and ENet. More importantly, DSNet highlights classes most critical with driving decision making thr…
This study investigates weakly supervised image segmentation using loose bounding box supervision. It presents a multiple instance learning strategy based on polar transformation to assist image segmentation when loose bounding boxes are employed as supervision. In this strategy, weighted smooth maximum approximation is introduced to incorporate the observation that pixels closer to the origin of the polar transformation are more likely to belong to the object in the bounding box. The proposed approach was evaluated on a public medical dataset using Dice coefficient. The results demonstrate it…
University College London(伦敦大学学院);KU Leuven(鲁汶大学);University Hospitals KU Leuven(鲁汶大学医院)
Convolutional neural networks (CNNs) have achieved state-of-the-art performance for automatic medical image segmentation. However, they have not demonstrated sufficiently accurate and robust results for clinical use. In addition, they are limited by the lack of image-specific adaptation and the lack of generalizability to previously unseen object classes. To address these problems, we propose a novel deep learning-based framework for interactive segmentation by incorporating CNNs into a bounding box and scribble-based segmentation pipeline. We propose image-specific fine-tuning to make a CNN m…
Segmentation is one of the most important tasks in MRI medical image analysis and is often the first and the most critical step in many clinical applications. In brain MRI analysis, head segmentation is commonly used for measuring and visualizing the brain's anatomical structures and is also a necessary step for other applications such as current-source reconstruction in electroencephalography and magnetoencephalography (EEG/MEG). Here we propose a deep learning framework that can segment brain, skull, and extra-cranial tissue using only T1-weighted MRI as input. In addition, we describe a rob…
Washington University School of Medicine(华盛顿大学医学院)
Deep Learning networks have established themselves as providing state of art performance for semantic segmentation. These techniques are widely applied specifically to medical detection, segmentation and classification. The advent of the U-Net based architecture has become particularly popular for this application. In this paper we present the Dense Recurrent Residual Convolutional Neural Network(Dense R2U CNN) which is a synthesis of Recurrent CNN, Residual Network and Dense Convolutional Network based on the U-Net model architecture. The residual unit helps training deeper network, while the…
University of Maryland at College Park(马里兰大学帕克分校);Raytheon BBN Technologies(雷神BBN技术公司)
A multi-scale greedy-based object proposal generation approach is presented. Based on the multi-scale nature of objects in images, our approach is built on top of a hierarchical segmentation. We first identify the representative and diverse exemplar clusters within each scale by using a diversity ranking algorithm. Object proposals are obtained by selecting a subset from the multi-scale segment pool via maximizing a submodular objective function, which consists of a weighted coverage term, a single-scale diversity term and a multi-scale reward term. The weighted coverage term forces the select…
We present a single network method for panoptic segmentation. This method combines the predictions from a jointly trained semantic and instance segmentation network using heuristics. Joint training is the first step towards an end-to-end panoptic segmentation network and is faster and more memory efficient than training and predicting with two networks, as done in previous work. The architecture consists of a ResNet-50 feature extractor shared by the semantic segmentation and instance segmentation branch. For instance segmentation, a Mask R-CNN type of architecture is used, while the semantic …
Graduate School at Shenzhen, Tsinghua University(清华大学深圳研究生院);The University of Texas at Arlington(德克萨斯大学阿灵顿分校)
Accurate segmentation of brain tissue in magnetic resonance images (MRI) is a diffcult task due to different types of brain abnormalities. Using information and features from multimodal MRI including T1, T1-weighted inversion recovery (T1-IR) and T2-FLAIR and differential geometric features including the Jacobian determinant(JD) and the curl vector(CV) derived from T1 modality can result in a more accurate analysis of brain images. In this paper, we use the differential geometric information including JD and CV as image characteristics to measure the differences between different MRI images, w…
Deep learning (DL) based semantic segmentation methods have been providing state-of-the-art performance in the last few years. More specifically, these techniques have been successfully applied to medical image classification, segmentation, and detection tasks. One deep learning technique, U-Net, has become one of the most popular for these applications. In this paper, we propose a Recurrent Convolutional Neural Network (RCNN) based on U-Net as well as a Recurrent Residual Convolutional Neural Network (RRCNN) based on U-Net models, which are named RU-Net and R2U-Net respectively. The proposed …
CLIP-VIS: Adapting CLIP for Open-Vocabulary Video Instance Segmentation
School of Electrical and Information Engineering, Tianjin University(天津大学电气与信息工程学院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);School of Big Data and Software Engineering, Chongqing University(重庆大学大数据与软件学院)
We propose a novel technique to incorporate attention within convolutional neural networks using feature maps generated by a separate convolutional autoencoder. Our attention architecture is well suited for incorporation with deep convolutional networks. We evaluate our model on benchmark segmentation datasets in skin cancer segmentation and lung lesion segmentation. Results show highly competitive performance when compared with U-Net and it's residual variant.
We present a simple and effective framework for simultaneous semantic segmentation and instance segmentation with Fully Convolutional Networks (FCNs). The method, called BiSeg, predicts instance segmentation as a posterior in Bayesian inference, where semantic segmentation is used as a prior. We extend the idea of position-sensitive score maps used in recent methods to a fusion of multiple score maps at different scales and partition modes, and adopt it as a robust likelihood for instance segmentation inference. As both Bayesian inference and map fusion are performed per pixel, BiSeg is a full…
Facebook AI Research(Facebook 人工智能研究院);INRIA(法国国家信息与自动化研究所);Laboratoire Jean Kuntzmann(让·昆兹曼实验室)
Adversarial training has been shown to produce state of the art results for generative image modeling. In this paper we propose an adversarial training approach to train semantic segmentation models. We train a convolutional semantic segmentation network along with an adversarial network that discriminates segmentation maps coming either from the ground truth or from the segmentation network. The motivation for our approach is that it can detect and correct higher-order inconsistencies between ground truth segmentation maps and the ones produced by the segmentation net. Our experiments show th…
Queen Mary University of London(伦敦玛丽女王大学);Karolinska University Hospital(卡罗林斯卡大学医院)
We present a novel neural network architecture, US-Net, for robust nuclei instance segmentation in histopathology images. The proposed framework integrates the nuclei detection and segmentation networks by sharing their outputs through the same foundation network, and thus enhancing the performance of both. The detection network takes into account the high-level semantic cues with contextual information, while the segmentation network focuses more on the low-level details like the edges. Extensive experiments reveal that our proposed framework can strengthen the performance of both branch netw…
We propose a novel deep neural network architecture for semi-supervised semantic segmentation using heterogeneous annotations. Contrary to existing approaches posing semantic segmentation as a single task of region-based classification, our algorithm decouples classification and segmentation, and learns a separate network for each task. In this architecture, labels associated with an image are identified by classification network, and binary segmentation is subsequently performed for each identified label in segmentation network. The decoupled architecture enables us to learn classification an…