In this work, we study the problem of training deep networks for semantic image segmentation using only a fraction of annotated images, which may significantly reduce human annotation efforts. Particularly, we propose a strategy that exploits the unpaired image style transfer capabilities of CycleGAN in semi-supervised segmentation. Unlike recent works using adversarial learning for semi-supervised segmentation, we enforce cycle consistency to learn a bidirectional mapping between unpaired images and segmentation masks. This adds an unsupervised regularization effect that boosts the segmentation performance when annotated data is limited. Experiments on three different public segmentation benchmarks (PASCAL VOC 2012, Cityscapes and ACDC) demonstrate the effectiveness of the proposed method. The proposed model achieves 2-4% of improvement with respect to the baseline and outperforms recent approaches for this task, particularly in low labeled data regime.
In recent years, semantic segmentation has taken benefit from various works in computer vision. Inspired by the very versatile CycleGAN architecture, we combine semantic segmentation with the concept of cycle consistency to enable a multitask training protocol. However, learning is largely prevented by the so-called steganography effect, which expresses itself as watermarks in the latent segmentation domain, making image reconstruction a too easy task. To combat this, we propose a noise injection, based either on quantization noise or on Gaussian noise addition to avoid this disadvantageous in…
The prevailing paradigm for producing semantic segmentation training data relies on densely labelling each pixel of each image in the training set, akin to colouring-in books. This approach becomes a bottleneck when scaling up in the number of images, classes, and annotators. Here we propose instead a pointillist approach for semantic segmentation annotation, where only point-wise yes/no questions are answered. We explore design alternatives for such an active learning approach, measure the speed and consistency of human annotators on this task, show that this strategy enables training good se…
Recent advances in generative models and adversarial training have led to a flourishing image-to-image (I2I) translation literature. The current I2I translation approaches require training images from the two domains that are either all paired (supervised) or all unpaired (unsupervised). In practice, obtaining paired training data in sufficient quantities is often very costly and cumbersome. Therefore solutions that employ unpaired data, while less accurate, are largely preferred. In this paper, we aim to bridge the gap between supervised and unsupervised I2I translation, with application to s…
University of Florida（佛罗里达大学）；University of Colorado Denver（科罗拉多大学丹佛分校）；Xiamen University（厦门大学）
In this paper, we propose a simple but effective method for fast image segmentation. We re-examine the locality-preserving character of spectral clustering by constructing a graph over image regions with both global and local connections. Our novel approach to build graph connections relies on two key observations: 1) local region pairs that co-occur frequently will have a high probability to reside on a common object; 2) spatially distant regions in a common object often exhibit similar visual saliency, which implies their neighborship in a manifold. We present a novel energy function to effi…
Australian Institute for Machine Learning（澳大利亚机器学习研究所）；University of Adelaide（阿德莱德大学）
Autoencoding, which aims to reconstruct the input images through a bottleneck latent representation, is one of the classic feature representation learning strategies. It has been shown effective as an auxiliary task for semi-supervised learning but has become less popular as more sophisticated methods have been proposed in recent years. In this paper, we revisit the idea of using image reconstruction as the auxiliary task and incorporate it with a modern semi-supervised semantic segmentation framework. Surprisingly, we discover that such an old idea in semi-supervised learning can produce resu…
Revisiting foundation models for cell instance segmentation
Georg-August-University Göttingen, Institute of Computer Science（哥廷根大学计算机科学学院）；CAIMed - Lower Saxony Center for AI & Causal Methods in Medicine, Göttingen（下萨克森人工智能与因果医学方法中心，哥廷根）；Cluster of Excellence Multiscale Bioimaging (MBExC), Georg-August-University Göttingen（多尺度生物成像（MBExC）集群，哥廷根大学）
UIUC（伊利诺伊大学厄巴纳-香槟分校）；NUS（新加坡国立大学）；IBM Thomas J. Watson Research Center（IBM托马斯·J·沃森研究中心）；Tencent AI Lab（腾讯人工智能实验室）
Despite the remarkable progress, weakly supervised segmentation approaches are still inferior to their fully supervised counterparts. We obverse the performance gap mainly comes from their limitation on learning to produce high-quality dense object localization maps from image-level supervision. To mitigate such a gap, we revisit the dilated convolution [1] and reveal how it can be utilized in a novel way to effectively overcome this critical limitation of weakly supervised segmentation approaches. Specifically, we find that varying dilation rates can effectively enlarge the receptive fields o…
Seg-CycleGAN : SAR-to-optical image translation guided by a downstream task
Gina Cody School of Engineering and Computer Science, Concordia University（康考迪亚大学吉娜·科迪工程与计算机科学学院）；School of Automation, Northwestern Polytechnical University（西北工业大学自动化学院）；School of Cybersecurity, Northwestern Polytechnical University（西北工业大学网络空间安全学院）
In this work, we revisit the weak-to-strong consistency framework, popularized by FixMatch from semi-supervised classification, where the prediction of a weakly perturbed image serves as supervision for its strongly perturbed version. Intriguingly, we observe that such a simple pipeline already achieves competitive results against recent advanced works, when transferred to our segmentation scenario. Its success heavily relies on the manual design of strong data augmentations, however, which may be limited and inadequate to explore a broader perturbation space. Motivated by this, we propose an …
SemSim: Revisiting Weak-to-Strong Consistency from a Semantic Similarity Perspective for Semi-supervised Medical Image Segmentation
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；College of Information Science and Engineering, Ritsumeikan University（立命馆大学信息科学与工程学院）
Recently, a variety of vision transformers have been developed as their capability of modeling long-range dependency. In current transformer-based backbones for medical image segmentation, convolutional layers were replaced with pure transformers, or transformers were added to the deepest encoder to learn global context. However, there are mainly two challenges in a scale-wise perspective: (1) intra-scale problem: the existing methods lacked in extracting local-global cues in each scale, which may impact the signal propagation of small objects; (2) inter-scale problem: the existing methods fai…
DZYNE Technologies（DZYNE技术公司）；Kitware, Inc.（Kitware公司）；University of Kentucky（肯塔基大学）
This work addresses the task of overhead image segmentation when auxiliary ground-level images are available. Recent work has shown that performing joint inference over these two modalities, often called near/remote sensing, can yield significant accuracy improvements. Extending this line of work, we introduce the concept of geospatial attention, a geometry-aware attention mechanism that explicitly considers the geospatial relationship between the pixels in a ground-level image and a geographic location. We propose an approach for computing geospatial attention that incorporates geometric feat…
University of Southern California（南加州大学）；Simon Fraser University（西蒙菲莎大学）；Oracle Labs（甲骨文实验室）
Semantic segmentation requires a detailed labeling of image pixels by object category. Information derived from local image patches is necessary to describe the detailed shape of individual objects. However, this information is ambiguous and can result in noisy labels. Global inference of image content can instead capture the general semantic concepts present. We advocate that holistic inference of image concepts provides valuable information for detailed pixel labeling. We propose a generic framework to leverage holistic information in the form of a LabelBank for pixel-level segmentation. We …
Revisiting Network Perturbation for Semi-Supervised Semantic Segmentation
College of Computer and Data Science（计算机与数据科学学院）；Fuzhou University（福州大学）；School of Computer and Big Data（计算机与大数据学院）；Minjiang University（闽江学院）；School of Computer Science and Mathematics（计算机科学与数学学院）；Fujian University of Technology（福建理工大学）
Harvard University（哈佛大学）；Technical University of Munich（慕尼黑工业大学）；MIT（麻省理工学院）；University of Cambridge（剑桥大学）；Donostia International Physics Center（多诺斯蒂亚国际物理中心）；University of the Basque Country（巴斯克大学）；Ikerbasque, Basque Foundation for Science（巴斯克科学基金会伊克巴斯克）；HHMI（霍华德·休斯医学研究所）；Boston College（波士顿学院）
Instance segmentation for unlabeled imaging modalities is a challenging but essential task as collecting expert annotation can be expensive and time-consuming. Existing works segment a new modality by either deploying a pre-trained model optimized on diverse training data or conducting domain translation and image segmentation as two independent steps. In this work, we propose a novel Cyclic Segmentation Generative Adversarial Network (CySGAN) that conducts image translation and instance segmentation jointly using a unified framework. Besides the CycleGAN losses for image translation and super…
It is commonly believed that high internal resolution combined with expensive operations (e.g. atrous convolutions) are necessary for accurate semantic segmentation, resulting in slow speed and large memory usage. In this paper, we question this belief and demonstrate that neither high internal resolution nor atrous convolutions are necessary. Our intuition is that although segmentation is a dense per-pixel prediction task, the semantics of each pixel often depend on both nearby neighbors and far-away context; therefore, a more powerful multi-scale feature fusion network plays a critical role.…
East China University of Science and Technology（华东理工大学）；Potsdam Institute for Climate Impact Research（波茨坦气候影响研究所）；Humboldt University of Berlin（柏林洪堡大学）
Semantic segmentation and depth completion are two challenging tasks in scene understanding, and they are widely used in robotics and autonomous driving. Although several works are proposed to jointly train these two tasks using some small modifications, like changing the last layer, the result of one task is not utilized to improve the performance of the other one despite that there are some similarities between these two tasks. In this paper, we propose multi-task generative adversarial networks (Multi-task GANs), which are not only competent in semantic segmentation and depth completion, bu…
VSITE College for Information Technologies（VSITE信息技术学院）
Semi-supervised learning an attractive technique in practical deployments of deep models since it relaxes the dependence on labeled data. It is especially important in the scope of dense prediction because pixel-level annotation requires significant effort. This paper considers semi-supervised algorithms that enforce consistent predictions over perturbed unlabeled inputs. We study the advantages of perturbing only one of the two model instances and preventing the backward pass through the unperturbed instance. We also propose a competitive perturbation model as a composition of geometric warp …
Deep learning highly relies on the quantity of annotated data. However, the annotations for 3D volumetric medical data require experienced physicians to spend hours or even days for investigation. Self-supervised learning is a potential solution to get rid of the strong requirement of training data by deeply exploiting raw data information. In this paper, we propose a novel self-supervised learning framework for volumetric medical images. Specifically, we propose a context restoration task, i.e., Rubik's cube++, to pre-train 3D neural networks. Different from the existing context-restoration-b…
High-resolution images enable neural networks to learn richer visual representations. However, this improved performance comes at the cost of growing computational complexity, hindering their usage in latency-sensitive applications. As not all pixels are equal, skipping computations for less-important regions offers a simple and effective measure to reduce the computation. This, however, is hard to be translated into actual speedup for CNNs since it breaks the regularity of the dense convolution workload. In this paper, we introduce SparseViT that revisits activation sparsity for recent window…
Revisiting Continual Semantic Segmentation with Pre-trained Vision Models
Mohamed bin Zayed University of Artificial Intelligence（莫扎德·本·扎耶德人工智能大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Shenyang Institute of Automation, Chinese Academy of Sciences（中国科学院沈阳自动化研究所）；South China University of Technology（华南理工大学）；Huawei Inc.（华为公司）；Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences（中国科学院脑科学与智能技术卓越创新中心）
Data Synthesis Improves 3D Myotube Instance Segmentation
Institute for Automation and Applied Informatics, Karlsruhe Institute of Technology（自动化与应用信息学院，卡尔斯鲁厄理工学院）；Institute of Biological and Chemical Systems, Karlsruhe Institute of Technology（生物与化学系统研究所，卡尔斯鲁厄理工学院）；CeMOS Research and Transfer Center, Technische Hochschule Mannheim（CeMOS研究与转移中心，曼海姆技术大学）
Medical University of Vienna（维也纳医科大学）；Medical University Vienna（维也纳医科大学）
Optical coherence tomography (OCT) has become the most important imaging modality in ophthalmology. A substantial amount of research has recently been devoted to the development of machine learning (ML) models for the identification and quantification of pathological features in OCT images. Among the several sources of variability the ML models have to deal with, a major factor is the acquisition device, which can limit the ML model's generalizability. In this paper, we propose to reduce the image variability across different OCT devices (Spectralis and Cirrus) by using CycleGAN, an unsupervis…
Contrastive self-supervised learning has outperformed supervised pretraining on many downstream tasks like segmentation and object detection. However, current methods are still primarily applied to curated datasets like ImageNet. In this paper, we first study how biases in the dataset affect existing methods. Our results show that current contrastive approaches work surprisingly well across: (i) object- versus scene-centric, (ii) uniform versus long-tailed and (iii) general versus domain-specific datasets. Second, given the generality of the approach, we try to realize further gains with minor…
Revisiting Evaluation Metrics for Semantic Segmentation: Optimization and Evaluation of Fine-grained Intersection over Union
KU Leuven（鲁汶大学）；Google（谷歌公司）；Google DeepMind（谷歌DeepMind）；University of Oxford（牛津大学）；EPFL（洛桑联邦理工学院）；ETH Zürich（苏黎世联邦理工学院）；INSAIT（INSAIT研究院）；Samsung Research（三星研究院）
Wuhan University（武汉大学）；Rensselaer Polytechnic Institute (RPI)（伦斯勒理工学院（RPI））
Medical image segmentation is a fundamental task in medical image analysis. Despite that deep convolutional neural networks have gained stellar performance in this challenging task, they typically rely on large labeled datasets, which have limited their extension to customized applications. By revisiting the superiority of atlas based segmentation methods, we present a new framework of One-pass aligned Atlas Set for Images Segmentation (OASIS). To address the problem of time-consuming iterative image registration used for atlas warping, the proposed method takes advantage of the power of deep …
Domain adaptation is of huge interest as labeling is an expensive and error-prone task, especially when labels are needed on pixel-level like in semantic segmentation. Therefore, one would like to be able to train neural networks on synthetic domains, where data is abundant and labels are precise. However, these models often perform poorly on out-of-domain images. To mitigate the shift in the input, image-to-image approaches can be used. Nevertheless, standard image-to-image approaches that bridge the domain of deployment with the synthetic training domain do not focus on the downstream task b…
Mobile U-ViT: Revisiting large kernel and U-shaped ViT for efficient medical image segmentation
University of Science and Technology of China（中国科学技术大学）；Suzhou Institute for Advanced Research, USTC（苏州先进研究院）；Shanghai Jiao Tong University（上海交通大学）；Harbin Institute of Technology（哈尔滨工业大学）；State Grid Hunan ElectricPower Corporation Limited Research Institute（国网湖南电力有限公司研究院）
This paper presents our contribution to the REFUGE challenge 2020. The challenge consisted of three tasks based on a dataset of retinal images: Segmentation of optic disc and cup, classification of glaucoma, and localization of fovea. We propose employing convolutional neural networks for all three tasks. Segmentation is performed using a U-Net, classification is performed by a pre-trained InceptionV3 network, and fovea detection is performed by employing stacked hour-glass for heatmap prediction. The challenge dataset contains images from three different data sources. To enhance performance, …
The University of Texas at El Paso（德克萨斯大学埃尔帕索分校）；Microsoft AI for Good Research Lab（微软益世AI研究实验室）；Georgia Institute of Technology（佐治亚理工学院）；Microsoft Research（微软研究院）
Normalization layers have been shown to improve convergence in deep neural networks, and even add useful inductive biases. In many vision applications the local spatial context of the features is important, but most common normalization schemes including Group Normalization (GN), Instance Normalization (IN), and Layer Normalization (LN) normalize over the entire spatial dimension of a feature. This can wash out important signals and degrade performance. For example, in applications that use satellite imagery, input images can be arbitrarily large; consequently, it is nonsensical to normalize o…
Improving Deformable Image Registration Accuracy through a Hybrid Similarity Metric and CycleGAN Based Auto-Segmentation
Emory University（埃默里大学）；Winship Cancer Institute（温希普癌症研究所）；Drexel University（德雷塞尔大学）；Massachusetts General Hospital（马萨诸塞州总医院）；Harvard Medical School（哈佛医学院）
Universidad de Buenos Aires（布宜诺斯艾利斯大学）；Universidad Nacional del Litoral（国立滨海大学）
Automatic segmentation of white matter hyperintensities in magnetic resonance images is of paramount clinical and research importance. Quantification of these lesions serve as a predictor for risk of stroke, dementia and mortality. During the last years, convolutional neural networks (CNN) specifically tailored for biomedical image segmentation have outperformed all previous techniques in this task. However, they are extremely data-dependent, and maintain a good performance only when data distribution between training and test datasets remains unchanged. When such distribution changes but we s…
Computer Vision Center, Universitat Autònoma de Barcelona（巴塞罗那自治大学计算机视觉中心）
We address the problem of image translation between domains or modalities for which no direct paired data is available (i.e. zero-pair translation). We propose mix and match networks, based on multiple encoders and decoders aligned in such a way that other encoder-decoder pairs can be composed at test time to perform unseen image translation tasks between domains or modalities for which explicit paired samples were not seen during training. We study the impact of autoencoders, side information and losses in improving the alignment and transferability of trained pairwise translation models to u…
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；Microsoft Research（微软研究院）
PointNet++ is one of the most influential neural architectures for point cloud understanding. Although the accuracy of PointNet++ has been largely surpassed by recent networks such as PointMLP and Point Transformer, we find that a large portion of the performance gain is due to improved training strategies, i.e. data augmentation and optimization techniques, and increased model sizes rather than architectural innovations. Thus, the full potential of PointNet++ has yet to be explored. In this work, we revisit the classical PointNet++ through a systematic study of model training and scaling stra…
University of Chinese Academy of Sciences（中国科学院大学）；Microsoft Research（微软研究院）
Masked image modeling has demonstrated great potential to eliminate the label-hungry problem of training large-scale vision Transformers, achieving impressive performance on various downstream tasks. In this work, we propose a unified view of masked image modeling after revisiting existing methods. Under the unified view, we introduce a simple yet effective method, termed as MaskDistill, which reconstructs normalized semantic features from teacher models at the masked positions, conditioning on corrupted input images. Experimental results on image classification and semantic segmentation show …
University of Toronto（多伦多大学）；Peking University（北京大学）；Xi’an Jiaotong-Liverpool University（西交利物浦大学）；University of Pennsylvania（宾夕法尼亚大学）；CUHK（香港中文大学）；Microsoft Research Asia（微软亚洲研究院）
