Rapid growth in the field of quantitative digital image analysis is paving the way for researchers to make precise measurements about objects in an image. To compute quantities from the image such as the density of compressed materials or the velocity of a shockwave, we must determine object boundaries. Images containing regions that each have a spatial trend in intensity are of particular interest. We present a supervised image segmentation method that incorporates spatial information to locate boundaries between regions with overlapping intensity histograms. The segmentation of a pixel is determined by comparing its intensity to distributions from local, nearby pixel intensities. Because of the statistical nature of the algorithm, we use maximum likelihood estimation theory to quantify uncertainty about each boundary. We demonstrate the success of this algorithm on a radiograph of a multicomponent cylinder and on an optical image of a laser-induced shockwave, and we provide final boundary locations with associated bands of uncertainty.
In this paper we tackle the problem of unsupervised domain adaptation for the task of semantic segmentation, where we attempt to transfer the knowledge learned upon synthetic datasets with ground-truth labels to real-world images without any annotation. With the hypothesis that the structural content of images is the most informative and decisive factor to semantic segmentation and can be readily shared across domains, we propose a Domain Invariant Structure Extraction (DISE) framework to disentangle images into domain-invariant structure and domain-specific texture representations, which can …
Australian Centre for Field Robotics（澳大利亚野外机器人学中心）；University of Sydney（悉尼大学）
Semantic segmentation using deep neural networks has been widely explored to generate high-level contextual information for autonomous vehicles. To acquire a complete $180^\circ$ semantic understanding of the forward surroundings, we propose to stitch semantic images from multiple cameras with varying orientations. However, previously trained semantic segmentation models showed unacceptable performance after significant changes to the camera orientations and the lighting conditions. To avoid time-consuming hand labeling, we explore and evaluate the use of data augmentation techniques, specific…
University of Maryland（马里兰大学）；UC Berkeley（加州大学伯克利分校）
Deep neural networks exhibit exceptional accuracy when they are trained and tested on the same data distributions. However, neural classifiers are often extremely brittle when confronted with domain shift---changes in the input distribution that occur over time. We present ACE, a framework for semantic segmentation that dynamically adapts to changing environments over the time. By aligning the distribution of labeled training data from the original source domain with the distribution of incoming data in a shifted domain, ACE synthesizes labeled training data for environments as it sees them. T…
German Cancer Research Center (DKFZ)（德国癌症研究中心（DKFZ））
The U-Net was presented in 2015. With its straight-forward and successful architecture it quickly evolved to a commonly used benchmark in medical image segmentation. The adaptation of the U-Net to novel problems, however, comprises several degrees of freedom regarding the exact architecture, preprocessing, training and inference. These choices are not independent of each other and substantially impact the overall performance. The present paper introduces the nnU-Net ('no-new-Net'), which refers to a robust and self-adapting framework on the basis of 2D and 3D vanilla U-Nets. We argue the stron…
Automatic segmentation of microscopy images is an important task in medical image processing and analysis. Nucleus detection is an important example of this task. Mask-RCNN is a recently proposed state-of-the-art algorithm for object detection, object localization, and object instance segmentation of natural images. In this paper we demonstrate that Mask-RCNN can be used to perform highly effective and efficient automatic segmentations of a wide range of microscopy images of cell nuclei, for a variety of cells acquired under a variety of conditions.
ASLseg: Adapting SAM in the Loop for Semi-supervised Liver Tumor Segmentation
Southern University of Science and Technology（南方科技大学）；The University of Hong Kong（香港大学）；Jiaxing Research Institute, Southern University of Science and Technology（南方科技大学嘉兴研究院）
Training models dedicated to semantic segmentation requires a large amount of pixel-wise annotated data. Due to their costly nature, these annotations might not be available for the task at hand. To alleviate this problem, unsupervised domain adaptation approaches aim at aligning the feature distributions between the labeled source and the unlabeled target data. While these strategies lead to noticeable improvements, their effectiveness remains limited. To guide the domain adaptation task more efficiently, previous works attempted to include human interactions in this process under the form of…
Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation
University of Oxford（牛津大学）；National University of Singapore（新加坡国立大学）；University of Alberta（阿尔伯塔大学）；A*STAR（新加坡科技研究局）；Carnegie Mellon University（卡内基梅隆大学）；Singapore Eye Research Institute（新加坡眼科研究所）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；University of Hongkong（香港大学）
本文提出Medical SAM Adapter（Med-SA），通过轻量适配技术将医学知识融入SAM，利用SD-Trans和HyP-Adpt适配2D模型至3D医学图像，在17个任务上以仅2%参数更新超越SOTA方法。
We propose an approach to domain adaptation for semantic segmentation that is both practical and highly accurate. In contrast to previous work, we abandon the use of computationally involved adversarial objectives, network ensembles and style transfer. Instead, we employ standard data augmentation techniques $-$ photometric noise, flipping and scaling $-$ and ensure consistency of the semantic predictions across these image transformations. We develop this principle in a lightweight self-supervised framework trained on co-evolving pseudo labels without the need for cumbersome extra training ro…
SAM Fails to Segment Anything? -- SAM-Adapter: Adapting SAM in Underperformed Scenes: Camouflage, Shadow, Medical Image Segmentation, and More
KOKONI, Moxin (Huzhou) Tech. Co., LTD（莫西（湖州）科技有限公司KOKONI）；Beihang University（北京航空航天大学）；Zhejiang University（浙江大学）；Huzhou University（湖州师范学院）；Singapore University of Technology and Design（新加坡科技设计大学）
Adapting SAM to Nuclei Instance Segmentation and Classification via Cooperative Fine-Grained Refinement
organization= College of Computer；Data Science , addressline= Fuzhou University , city= Fuzhou , country= China；organization= School of Mechanical Engineering；Automation , addressline= Fuzhou University , city= Fuzhou , country= China；organization= College of Physics；Information Engineering , addressline= Fuzhou University , city= Fuzhou , country= China
VFM-ISRefiner: Towards Better Adapting Vision Foundation Models for Interactive Segmentation of Remote Sensing Images
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院航天信息研究所）；School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences（中国科学院大学电子电气与通信工程学院）；School of Information Science and Technology, Beijing Forestry University（北京林业大学信息科学与技术学院）
In this paper, we present the PS^2-Net -- a locally and globally aware deep learning framework for semantic segmentation on 3D scene-level point clouds. In order to deeply incorporate local structures and global context to support 3D scene segmentation, our network is built on four repeatedly stacked encoders, where each encoder has two basic components: EdgeConv that captures local structures and NetVLAD that models global context. Different from existing start-of-the-art methods for point-based scene semantic segmentation that either violate or do not achieve permutation invariance, our PS^2…
SAM2-Adapter: Evaluating & Adapting Segment Anything 2 in Downstream Tasks: Camouflage, Shadow, Medical Image Segmentation, and More
KOKONI, Moxin (Huzhou) Tech. Co., LTD（科科尼（湖州）魔芯科技有限公司）；School of Information Engineering, Huzhou University（湖州大学信息工程学院）；School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学技术学院）；School of Software Technology, Zhejiang University（浙江大学软件学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Information Systems Technology and Design Pillar, Singapore University of Technology and Design（新加坡科技设计大学信息系统技术与设计支柱学院）
CLIP-VIS: Adapting CLIP for Open-Vocabulary Video Instance Segmentation
School of Electrical and Information Engineering, Tianjin University（天津大学电气与信息工程学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；School of Big Data and Software Engineering, Chongqing University（重庆大学大数据与软件学院）
Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；Facebook Artificial Intelligence（脸书人工智能研究院）
Unsupervised domain adaptation (UDA) aims to transfer knowledge learned from a labeled source domain to an unlabeled and unseen target domain, which is usually trained on data from both domains. Access to the source domain data at the adaptation stage, however, is often limited, due to data storage or privacy issues. To alleviate this, in this work, we target source free UDA for segmentation, and propose to adapt an ``off-the-shelf" segmentation model pre-trained in the source domain to the target domain, with an adaptive batch-wise normalization statistics adaptation framework. Specifically, …
Exploiting synthetic data to learn deep models has attracted increasing attention in recent years. However, the intrinsic domain difference between synthetic and real images usually causes a significant performance drop when applying the learned model to real world scenarios. This is mainly due to two reasons: 1) the model overfits to synthetic images, making the convolutional filters incompetent to extract informative representation for real images; 2) there is a distribution difference between synthetic and real data, which is also known as the domain adaptation problem. To this end, we prop…
Convolutional neural networks (CNNs) have been widely used in various vision tasks, e.g. image classification, semantic segmentation, etc. Unfortunately, standard 2D CNNs are not well suited for spherical signals such as panorama images or spherical projections, as the sphere is an unstructured grid. In this paper, we present Spherical Transformer which can transform spherical signals into vectors that can be directly processed by standard CNNs such that many well-designed CNNs architectures can be reused across tasks and datasets by pretraining. To this end, the proposed method first uses loc…
University of Science and Technology of China（中国科学技术大学）；Huawei Inc.（华为公司）
We study unsupervised domain adaptation (UDA) for semantic segmentation. Currently, a popular UDA framework lies in self-training which endows the model with two-fold abilities: (i) learning reliable semantics from the labeled images in the source domain, and (ii) adapting to the target domain via generating pseudo labels on the unlabeled images. We find that, by decreasing/increasing the proportion of training samples from the target domain, the 'learning ability' is strengthened/weakened while the 'adapting ability' goes in the opposite direction, implying a conflict between these two abilit…
ETH Zurich（苏黎世联邦理工学院）；MPI for Informatics（马克斯·普朗克信息学研究所）；KU Leuven（鲁汶大学）
Traditional domain adaptive semantic segmentation addresses the task of adapting a model to a novel target domain under limited or no additional supervision. While tackling the input domain gap, the standard domain adaptation settings assume no domain change in the output space. In semantic prediction tasks, different datasets are often labeled according to different semantic taxonomies. In many real-world settings, the target domain task requires a different taxonomy than the one imposed by the source domain. We therefore introduce the more general taxonomy adaptive cross-domain semantic segm…
Adapting Medical Vision Foundation Models for Volumetric Medical Image Segmentation via Active Learning and Selective Semi-supervised Fine-tuning
Mallinckrodt Institute of Radiology（马林克罗德放射医学研究所）；Institute for Informatics, Data Science and Biostatistics（信息学、数据科学与生物统计学研究所）；Department of Radiology, Perelman School of Medicine（放射科，佩尔曼医学院）；Center for AI and Data Science for Integrated Diagnostics (AI2D)（集成诊断人工智能与数据科学中心（AI2D））
Domain adaptation (DA) has drawn high interests for its capacity to adapt a model trained on labeled source data to perform well on unlabeled or weakly labeled target data from a different domain. Most common DA techniques require the concurrent access to the input images of both the source and target domains. However, in practice, it is common that the source images are not available in the adaptation phase. This is a very frequent DA scenario in medical imaging, for instance, when the source and target images come from different clinical sites. We propose a novel formulation for adapting seg…
MS-DKC: A Dataset Knowledge Card Framework for Designing and Adapting Medical Image Segmentation Models
Center of Excellence in Precision Medicine and Digital Health, Faculty of Dentistry, Chulalongkorn University, Bangkok, Thailand（精准医学与数字健康中心，朱拉隆功大学牙科学院，泰国曼谷）；Department of Computer Engineering, COMSATS University Islamabad, Islamabad, Pakistan（计算机工程系，COMSATS伊斯兰堡大学，巴基斯坦伊斯兰堡）；School of Biomedical Engineering, UNSW, Sydney, NSW, Australia（生物医学工程学院，新南威尔士大学，澳大利亚悉尼，新南威尔士）；Visiting Scholar (Collaborative Projects), Center of Excellence in Precision Medicine and Digital Health, Chulalongkorn University, Bangkok, Thailand（访问学者（合作项目），精准医学与数字健康中心，朱拉隆功大学，泰国曼谷）；Department of Computing, Abasyn University Islamabad Campus (AUIC), Islamabad, Pakistan（计算系，阿巴斯扬大学伊斯兰堡校区（AUIC），巴基斯坦伊斯兰堡）；Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates（Mohamed bin Zayed人工智能大学，阿布扎比，阿拉伯联合酋长国）；College of Computer and Information Sciences, prince Sultan University, Riyadh, SAudi Arabia（计算机与信息科学学院，苏丹王子大学，沙特阿拉伯利雅得）
Due to the complicated nanoscale structures of current integrated circuits(IC) builds and low error tolerance of IC image segmentation tasks, most existing automated IC image segmentation approaches require human experts for visual inspection to ensure correctness, which is one of the major bottlenecks in large-scale industrial applications. In this paper, we present the first data-driven automatic error detection approach targeting two types of IC segmentation errors: wire errors and via errors. On an IC image dataset collected from real industry, we demonstrate that, by adapting existing CNN…
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；ByteDance Inc.（字节跳动公司）；Carl Zeiss AG（卡尔·蔡司股份公司）
Panoramic images with their 360-degree directional view encompass exhaustive information about the surrounding space, providing a rich foundation for scene understanding. To unfold this potential in the form of robust panoramic segmentation models, large quantities of expensive, pixel-wise annotations are crucial for success. Such annotations are available, but predominantly for narrow-angle, pinhole-camera images which, off the shelf, serve as sub-optimal resources for training panoramic models. Distortions and the distinct image-feature distribution in 360-degree panoramas impede the transfe…
SAGE: Shape-Adapting Gated Experts for Adaptive Histopathology Image Segmentation
University of Science, VNU-HCM（越南国家大学科学学院）；Trivita AI；University of Technology, VNU-HCM（越南国家大学技术学院）；Michigan State University, USA（美国密歇根州立大学）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
Beijing NetPosa Technologies Co., Ltd.（北京网罗科技有限公司）；Institut National des Sciences Appliquées de Lyon（里昂应用科学学院）
We propose a novel locally adaptive learning estimator for enhancing the inter- and intra- discriminative capabilities of Deep Neural Networks, which can be used as improved loss layer for semantic image segmentation tasks. Most loss layers compute pixel-wise cost between feature maps and ground truths, ignoring spatial layouts and interactions between neighboring pixels with same object category, and thus networks cannot be effectively sensitive to intra-class connections. Stride by stride, our method firstly conducts adaptive pooling filter operating over predicted feature maps, aiming to me…
Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentation
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Hunan University（湖南大学）；University of Oxford（牛津大学）；Zhejiang University（浙江大学）；ByteDance Inc.（字节跳动公司）；Beihang University（北京航空航天大学）
EndoARSS: Adapting Spatially-Aware Foundation Model for Efficient Activity Recognition and Semantic Segmentation in Endoscopic Surgery
Department of Electronic Engineering, The Chinese University of Hong Kong（香港中文大学电子工程系）；Shenzhen Research Institute, The Chinese University of Hong Kong（香港中文大学深圳研究院）；School of Advanced Manufacturing, Fuzhou University（福州大学先进制造学院）
Dense pixel-wise classification maps output by deep neural networks are of extreme importance for scene understanding. However, these maps are often partially inaccurate due to a variety of possible factors. Therefore, we propose to interactively refine them within a framework named DISCA (Deep Image Segmentation with Continual Adaptation). It consists of continually adapting a neural network to a target image using an interactive learning process with sparse user annotations as ground-truth. We show through experiments on three datasets using synthesized annotations the benefits of the approa…
Among the neural network compression techniques, knowledge distillation is an effective one which forces a simpler student network to mimic the output of a larger teacher network. However, most of such model distillation methods focus on the image-level classification task. Directly adapting these methods to the task of semantic segmentation only brings marginal improvements. In this paper, we propose a simple, yet effective knowledge representation referred to as pixel-wise feature similarities (PFS) to tackle the challenging distillation problem of semantic segmentation. The developed PFS en…
Huazhong University of Science and Technology（华中科技大学）；Alibaba Group（阿里巴巴集团）
Despite their success for semantic segmentation, convolutional neural networks are ill-equipped for incremental learning, \ie, adapting the original segmentation model as new classes are available but the initial training data is not retained. Actually, they are vulnerable to catastrophic forgetting problem. We try to address this issue by "inverting" the trained segmentation network to synthesize input images starting from random noise. To avoid setting detailed pixel-wise segmentation maps as the supervision manually, we propose the SegInversion to synthesize images using the image-level lab…
Télécom-Paris, Intitute Polytechnique de Paris（巴黎理工学院电信巴黎）；Shanghai Jiao Tong University（上海交通大学）；University of Trento（特伦托大学）；Fondazione Bruno Kessler（布鲁诺·凯塞勒基金会）
In this work we address multi-target domain adaptation (MTDA) in semantic segmentation, which consists in adapting a single model from an annotated source dataset to multiple unannotated target datasets that differ in their underlying data distributions. To address MTDA, we propose a self-training strategy that employs pseudo-labels to induce cooperation among multiple domain-specific classifiers. We employ feature stylization as an efficient way to generate image views that forms an integral part of self-training. Additionally, to prevent the network from overfitting to noisy pseudo-labels, w…
Data-driven sensor interpretation in autonomous driving can lead to highly implausible predictions as can most of the time be verified with common-sense knowledge. However, learning common knowledge only from data is hard and approaches for knowledge integration are an active research area. We propose to use a partly human-designed, partly learned set of rules to describe relations between objects of a traffic scene on a high level of abstraction. In doing so, we improve and robustify existing deep neural networks consuming low-level sensor information. We present an initial study adapting the…
The Chinese University of Hong Kong（香港中文大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；The University of Hong Kong（香港大学）；SmartMore Corporation（SmartMore 公司）
Semantic segmentation is still a challenging task for parsing diverse contexts in different scenes, thus the fixed classifier might not be able to well address varying feature distributions during testing. Different from the mainstream literature where the efficacy of strong backbones and effective decoder heads has been well studied, in this paper, additional contextual hints are instead exploited via learning a context-aware classifier whose content is data-conditioned, decently adapting to different latent distributions. Since only the classifier is dynamically altered, our method is model-…
Unsupervised domain adaptation enables to alleviate the need for pixel-wise annotation in the semantic segmentation. One of the most common strategies is to translate images from the source domain to the target domain and then align their marginal distributions in the feature space using adversarial learning. However, source-to-target translation enlarges the bias in translated images and introduces extra computations, owing to the dominant data size of the source domain. Furthermore, consistency of the joint distribution in source and target domains cannot be guaranteed through global feature…
