This paper presents an end-to-end instance segmentation framework, termed SOIT, that Segments Objects with Instance-aware Transformers. Inspired by DETR \cite{carion2020end}, our method views instance segmentation as a direct set prediction problem and effectively removes the need for many hand-crafted components like RoI cropping, one-to-many label assignment, and non-maximum suppression (NMS). In SOIT, multiple queries are learned to directly reason a set of object embeddings of semantic category, bounding-box location, and pixel-wise mask in parallel under the global image context. The class and bounding-box can be easily embedded by a fixed-length vector. The pixel-wise mask, especially, is embedded by a group of parameters to construct a lightweight instance-aware transformer. Afterward, a full-resolution mask is produced by the instance-aware transformer without involving any RoI-based operation. Overall, SOIT introduces a simple single-stage instance segmentation framework that is both RoI- and NMS-free. Experimental results on the MS COCO dataset demonstrate that SOIT outperforms state-of-the-art instance segmentation approaches significantly. Moreover, the joint learning of multiple tasks in a unified query embedding can also substantially improve the detection performance. Code is available at \url{https://github.com/yuxiaodongHRI/SOIT}.
China Agricultural University（中国农业大学）；University of California, Berkeley（加州大学伯克利分校）
Most recent transformer-based models show impressive performance on vision tasks, even better than Convolution Neural Networks (CNN). In this work, we present a novel, flexible, and effective transformer-based model for high-quality instance segmentation. The proposed method, Segmenting Objects with TRansformers (SOTR), simplifies the segmentation pipeline, building on an alternative CNN backbone appended with two parallel subtasks: (1) predicting per-instance category via transformer and (2) dynamically generating segmentation mask with the multi-level upsampling module. SOTR can effectively …
In this paper, we propose an end-to-end framework for instance segmentation. Based on the recently introduced DETR [1], our method, termed SOLQ, segments objects by learning unified queries. In SOLQ, each query represents one object and has multiple representations: class, location and mask. The object queries learned perform classification, box regression and mask encoding simultaneously in an unified vector form. During training phase, the mask vectors encoded are supervised by the compression coding of raw spatial masks. In inference time, mask vectors produced can be directly transformed t…
Due to the fact that fully supervised semantic segmentation methods require sufficient fully-labeled data to work well and can not generalize to unseen classes, few-shot segmentation has attracted lots of research attention. Previous arts extract features from support and query images, which are processed jointly before making predictions on query images. The whole process is based on convolutional neural networks (CNN), leading to the problem that only local information is used. In this paper, we propose a TRansformer-based Few-shot Semantic segmentation method (TRFS). Specifically, our model…
Detecting and segmenting object instances is a common task in biomedical applications. Examples range from detecting lesions on functional magnetic resonance images, to the detection of tumours in histopathological images and extracting quantitative single-cell information from microscopy imagery, where cell segmentation is a major bottleneck. Attention-based transformers are state-of-the-art in a range of deep learning fields. They have recently been proposed for segmentation tasks where they are beginning to outperforming other methods. We present a novel attention-based cell detection trans…
TopoMamba: Topology-Aware Scanning and Fusion for Segmenting Heterogeneous Medical Visual Media
Dalian University of Technology（大连理工大学）；Guangdong University of Technology（广东工业大学）；Huizhou University（惠州市学院）；University of Macau（澳门大学）；Nanjing University of Posts and Telecommunications（南京邮电大学）；Hong Kong Polytechnic University（香港理工大学）
Video instance segmentation (VIS) is the task that requires simultaneously classifying, segmenting and tracking object instances of interest in video. Recent methods typically develop sophisticated pipelines to tackle this task. Here, we propose a new video instance segmentation framework built upon Transformers, termed VisTR, which views the VIS task as a direct end-to-end parallel sequence decoding/prediction problem. Given a video clip consisting of multiple image frames as input, VisTR outputs the sequence of masks for each instance in the video in order directly. At the core is a new, eff…
Moving objects have special importance for Autonomous Driving tasks. Detecting moving objects can be posed as Moving Object Segmentation, by segmenting the object pixels, or Moving Object Detection, by generating a bounding box for the moving targets. In this paper, we present a Multi-Task Learning architecture, based on Transformers, to jointly perform both tasks through one network. Due to the importance of the motion features to the task, the whole setup is based on a Spatio-Temporal aggregation. We evaluate the performance of the individual tasks architecture versus the MTL setup, both wit…
PLA Strategic Force Information Engineering University（中国人民解放军战略支援部队信息工程大学）；Space Engineering University（航天工程大学）；State Key Laboratory of Geo-Information Engineering（地理信息工程国家重点实验室）；Beijing University of Technology（北京工业大学）；University of Trento（特伦托大学）；Beijing Institute of Remote Sensing Information（北京遥感信息研究所）；China University of Geosciences (Beijing)（中国地质大学（北京））；ETH Zurich（苏黎世联邦理工学院）
Long-range contextual information is crucial for the semantic segmentation of High-Resolution (HR) Remote Sensing Images (RSIs). However, image cropping operations, commonly used for training neural networks, limit the perception of long-range contexts in large RSIs. To overcome this limitation, we propose a Wide-Context Network (WiCoNet) for the semantic segmentation of HR RSIs. Apart from extracting local features with a conventional CNN, the WiCoNet has an extra context branch to aggregate information from a larger image area. Moreover, we introduce a Context Transformer to embed contextual…
Segmenting Low-Contrast XCTs of Concrete: An Unsupervised Approach
Department of Mechanics, Faculty of Civil Engineering, Czech Technical University in Prague（机械系，土木工程学院，布拉格捷克技术大学）；Structural Composites Unit, Luxembourg Institute of Science and Technology (LIST)（结构复合材料单元，卢森堡科学与技术研究所（LIST））
The University of Hong Kong（香港大学）；Sensetime Research（商汤科技研究院）；Nanjing University（南京大学）
This work presents a new fine-grained transparent object segmentation dataset, termed Trans10K-v2, extending Trans10K-v1, the first large-scale transparent object segmentation dataset. Unlike Trans10K-v1 that only has two limited categories, our new dataset has several appealing benefits. (1) It has 11 fine-grained categories of transparent objects, commonly occurring in the human domestic environment, making it more practical for real-world application. (2) Trans10K-v2 brings more challenges for the current advanced segmentation methods than its former version. Furthermore, a novel transforme…
A Recent Survey of Vision Transformers for Medical Image Segmentation
Pakistan Institute of Engineering & Applied Sciences（巴基斯坦工程与应用科学研究所）；Avid Radio pharmaceuticals（Avid放射性制药公司）；Eli Lilly and Company（礼来公司）；University of Engineering and Applied Sciences（工程与应用科学大学）；Air University（航空大学）；Islamia College Peshawar（白沙瓦伊斯兰学院）；Korea National University of Transportation（韩国国立交通大学）
University at Buffalo（布法罗大学）；OPPO US Research Center, InnoPeak Technology Inc.（OPPO美国研究中心，InnoPeak科技公司）
Video instance segmentation (VIS) task requires classifying, segmenting, and tracking object instances over all frames in a video clip. Recently, VisTR has been proposed as end-to-end transformer-based VIS framework, while demonstrating state-of-the-art performance. However, VisTR is slow to converge during training, requiring around 1000 GPU hours due to the high computational cost of its transformer attention module. To improve the training efficiency, we propose Deformable VisTR, leveraging spatio-temporal deformable attention module that only attends to a small fixed set of key spatio-temp…
We propose a novel transformer model, capable of segmenting medical images of varying modalities. Challenges posed by the fine grained nature of medical image analysis mean that the adaptation of the transformer for their analysis is still at nascent stages. The overwhelming success of the UNet lay in its ability to appreciate the fine-grained nature of the segmentation task, an ability which existing transformer based models do not currently posses. To address this shortcoming, we propose The Fully Convolutional Transformer (FCT), which builds on the proven ability of Convolutional Neural Net…
Iran University of Science and Technology（伊朗科技大学）；RWTH Aachen University（亚琛工业大学）；Shahid Beheshti University（沙希德·贝赫什提大学）；University of Montreal（蒙特利尔大学）；Polytechnique Montreal（蒙特利尔理工学院）；Quebec AI Institute（魁北克人工智能研究所）；University of Regensburg（雷根斯堡大学）；Fraunhofer Institute for Digital Medicine MEVIS（弗劳恩霍夫数字医学MEVIS研究所）
Convolutional neural networks (CNNs) have been the consensus for medical image segmentation tasks. However, they suffer from the limitation in modeling long-range dependencies and spatial correlations due to the nature of convolution operation. Although transformers were first developed to address this issue, they fail to capture low-level features. In contrast, it is demonstrated that both local and global features are crucial for dense prediction, such as segmenting in challenging contexts. In this paper, we propose HiFormer, a novel method that efficiently bridges a CNN and a transformer fo…
Historical maps provide useful spatio-temporal information on the Earth's surface before modern earth observation techniques came into being. To extract information from maps, neural networks, which gain wide popularity in recent years, have replaced hand-crafted map processing methods and tedious manual labor. However, aleatoric uncertainty, known as data-dependent uncertainty, inherent in the drawing/scanning/fading defects of the original map sheets and inadequate contexts when cropping maps into small tiles considering the memory limits of the training process, challenges the model to make…
Axial Attention Transformer Networks: A New Frontier in Breast Cancer Detection
University of California, Los Angeles（加州大学洛杉矶分校）；Washington University in St. Louis（华盛顿大学圣路易斯分校）；Johns Hopkins University（约翰斯·霍普金斯大学）；Boston University（波士顿大学）；Northeastern University（东北大学）；Tulane University（杜兰大学）
A Simple yet Effective Network based on Vision Transformer for Camouflaged Object and Salient Object Detection
School of Electrical and Information Engineering, Tianjin University（天津大学电气与信息工程学院）；School of Computing and Information Technology, Great Bay University（湾区大学计算与信息技术学院）；Computer Vision and Pattern Recognition Laboratory, Lappeenranta-Lahti University of Technology LUT（拉彭兰塔-拉赫蒂理工大学计算机视觉与模式识别实验室）；School of Statistics and Data Science, Nankai University（南开大学统计与数据科学学院）
本文提出基于 ViT 的简单非对称编解码网络 SENet，结合 LICM 与动态加权损失，并解决 COD/SOD 联合训练冲突，在两类检测任务上均取得有竞争力的性能。
Low-Rank Continual Pyramid Vision Transformer: Incrementally Segment Whole-Body Organs in CT with Light-Weighted Adaptation
DAMO Academy, Alibaba Group（阿里巴巴达摩院）；State University of New York at Stony Brook（纽约州立大学石溪分校）；Johns Hopkins University（约翰斯·霍普金斯大学）；The First Affiliated Hospital Zhejiang University（浙江大学医学院附属第一医院）
RWTH Aachen University（亚琛工业大学）；Iran University of Science and Technology（伊朗科技大学）；Tehran University of Medical Sciences（德黑兰医科大学）；Shahid Beheshti University（沙希德·贝赫什提大学）；Stanford University（斯坦福大学）；Fraunhofer Institute for Digital Medicine MEVIS（弗劳恩霍夫数字医学研究所MEVIS）
Convolutional neural networks (CNNs) have been the de facto standard in a diverse set of computer vision tasks for many years. Especially, deep neural networks based on seminal architectures such as U-shaped models with skip-connections or atrous convolution with pyramid pooling have been tailored to a wide range of medical image analysis tasks. The main advantage of such architectures is that they are prone to detaining versatile local features. However, as a general consensus, CNNs fail to capture long-range dependencies and spatial correlations due to the intrinsic property of confined rece…
GLoG-CSUnet: Enhancing Vision Transformers with Adaptable Radiomic Features for Medical Image Segmentation
Computer Science Department Michigan State University East Lansing, USA（密歇根州立大学计算机科学系）；Department of Radiation Oncology Henry Ford Health Detroit, US（亨利福特健康辐射肿瘤科）；Division of Engineering New York University Abu Dhabi, UAE（纽约大学工程学院）
Microsoft Research Asia（微软亚洲研究院）；Carnegie Mellon University（卡内基梅隆大学）；Huazhong University of Science and Technology（华中科技大学）
Video instance segmentation (VIS) aims at classifying, segmenting and tracking object instances in video sequences. Recent transformer-based neural networks have demonstrated their powerful capability of modeling spatio-temporal correlations for the VIS task. Relying on video- or clip-level input, they suffer from high latency and computational cost. We propose a robust context fusion network to tackle VIS in an online fashion, which predicts instance segmentation frame-by-frame with a few preceding frames. To acquire the precise and temporal-consistent prediction for each frame efficiently, t…
TokenCut: Segmenting Objects in Images and Videos with Self-supervised Transformer and Normalized Cut
MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；LIGM (UMR 8049) - Ecole des Ponts, UPE（巴黎东部大学马恩拉瓦莱计算机实验室（巴黎桥路学校））；Tencent AI Lab（腾讯人工智能实验室）；Samsung AI Center, Cambridge（三星剑桥人工智能中心）；Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG（格勒诺布尔阿尔卑斯大学、法国国家科学研究中心、格勒诺布尔国立理工学院、格勒诺布尔信息学实验室）
The Hong Kong University of Science and Technology（香港科技大学）；Kuaishou Technology（快手科技）
Segmenting highly-overlapping image objects is challenging, because there is typically no distinction between real object contours and occlusion boundaries on images. Unlike previous instance segmentation methods, we model image formation as a composition of two overlapping layers, and propose Bilayer Convolutional Network (BCNet), where the top layer detects occluding objects (occluders) and the bottom layer infers partially occluded instances (occludees). The explicit modeling of occlusion relationship with bilayer structure naturally decouples the boundaries of both the occluding and occlud…
University of Oxford（牛津大学）；Shanghai Jiao Tong University（上海交通大学）
The objective of this paper is a model that is able to discover, track and segment multiple moving objects in a video. We make four contributions: First, we introduce an object-centric segmentation model with a depth-ordered layer representation. This is implemented using a variant of the transformer architecture that ingests optical flow, where each query vector specifies an object and its layer for the entire video. The model can effectively discover multiple moving objects and handle mutual occlusions; Second, we introduce a scalable pipeline for generating multi-object synthetic training d…
The Hong Kong University of Science and Technology（香港科技大学）；ETH Zürich（苏黎世联邦理工学院）；Kuaishou Technology（快手科技）
While Video Instance Segmentation (VIS) has seen rapid progress, current approaches struggle to predict high-quality masks with accurate boundary details. Moreover, the predicted segmentations often fluctuate over time, suggesting that temporal consistency cues are neglected or not fully utilized. In this paper, we set out to tackle these issues, with the aim of achieving highly detailed and more temporally stable mask predictions for VIS. We first propose the Video Mask Transfiner (VMT) method, capable of leveraging fine-grained high-resolution features thanks to a highly efficient video tran…
AFFSegNet: Adaptive Feature Fusion Segmentation Network for Microtumors and Multi-Organ Segmentation
Southern University of Science and Technology（南方科技大学）；Nanjing University of Posts and Telecommunications（南京邮电大学）；University of Macau（澳门大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
Self-supervised prediction is a powerful mechanism to learn representations that capture the underlying structure of the data. Despite recent progress, the self-supervised video prediction task is still challenging. One of the critical factors that make the task hard is motion segmentation, which is segmenting individual objects and the background and estimating their motion separately. In video prediction, the shape, appearance, and transformation of each object should be understood only by predicting the next frame in pixel space. To address this task, we propose a novel end-to-end learnable…
Common fully glazed facades and transparent objects present architectural barriers and impede the mobility of people with low vision or blindness, for instance, a path detected behind a glass door is inaccessible unless it is correctly perceived and reacted. However, segmenting these safety-critical objects is rarely covered by conventional assistive technologies. To tackle this issue, we construct a wearable system with a novel dual-head Transformer for Transparency (Trans4Trans) model, which is capable of segmenting general and transparent objects and performing real-time wayfinding to assis…
