The "Roaring 20s" of visual recognition began with the introduction of Vision Transformers (ViTs), which quickly superseded ConvNets as the state-of-the-art image classification model. A vanilla ViT, on the other hand, faces difficulties when applied to general computer vision tasks such as object detection and semantic segmentation. It is the hierarchical Transformers (e.g., Swin Transformers) that reintroduced several ConvNet priors, making Transformers practically viable as a generic vision backbone and demonstrating remarkable performance on a wide variety of vision tasks. However, the effectiveness of such hybrid approaches is still largely credited to the intrinsic superiority of Transformers, rather than the inherent inductive biases of convolutions. In this work, we reexamine the design spaces and test the limits of what a pure ConvNet can achieve. We gradually "modernize" a standard ResNet toward the design of a vision Transformer, and discover several key components that contribute to the performance difference along the way. The outcome of this exploration is a family of pure ConvNet models dubbed ConvNeXt. Constructed entirely from standard ConvNet modules, ConvNeXts compete favorably with Transformers in terms of accuracy and scalability, achieving 87.8% ImageNet top-1 accuracy and outperforming Swin Transformers on COCO detection and ADE20K segmentation, while maintaining the simplicity and efficiency of standard ConvNets.
The recent 3D medical ViTs (e.g., SwinUNETR) achieve the state-of-the-art performances on several 3D volumetric data benchmarks, including 3D medical image segmentation. Hierarchical transformers (e.g., Swin Transformers) reintroduced several ConvNet priors and further enhanced the practical viability of adapting volumetric segmentation in 3D medical datasets. The effectiveness of hybrid approaches is largely credited to the large receptive field for non-local self-attention and the large number of model parameters. In this work, we propose a lightweight volumetric ConvNet, termed 3D UX-Net, w…
We present a two-module approach to semantic segmentation that incorporates Convolutional Networks (CNNs) and Graphical Models. Graphical models are used to generate a small (5-30) set of diverse segmentations proposals, such that this set has high recall. Since the number of required proposals is so low, we can extract fairly complex features to rank them. Our complex feature of choice is a novel CNN called SegNet, which directly outputs a (coarse) semantic segmentation. Importantly, SegNet is specifically trained to optimize the corpus-level PASCAL IOU loss function. To the best of our knowl…
This paper does not attempt to design a state-of-the-art method for visual recognition but investigates a more efficient way to make use of convolutions to encode spatial features. By comparing the design principles of the recent convolutional neural networks ConvNets) and Vision Transformers, we propose to simplify the self-attention by leveraging a convolutional modulation operation. We show that such a simple approach can better take advantage of the large kernels (>=7x7) nested in convolutional layers. We build a family of hierarchical ConvNets using the proposed convolutional modulation, …
The Chinese University of Hong Kong（香港中文大学）；University of Michigan（密歇根大学）；Imsight Medical Technology Inc.（Imsight医疗科技公司）
Convolutional networks (ConvNets) have achieved great successes in various challenging vision tasks. However, the performance of ConvNets would degrade when encountering the domain shift. The domain adaptation is more significant while challenging in the field of biomedical image analysis, where cross-modality data have largely different distributions. Given that annotating the medical data is especially expensive, the supervised transfer learning approaches are not quite optimal. In this paper, we propose an unsupervised domain adaptation framework with adversarial learning for cross-modality…
Recent years have seen tremendous progress in still-image segmentation; however the naïve application of these state-of-the-art algorithms to every video frame requires considerable computation and ignores the temporal continuity inherent in video. We propose a video recognition framework that relies on two key observations: 1) while pixels may change rapidly from frame to frame, the semantic content of a scene evolves more slowly, and 2) execution can be viewed as an aspect of architecture, yielding purpose-fit computation schedules for networks. We define a novel family of "clockwork" convne…
Heidelberg University（海德堡大学）；Harvard Medical School（哈佛医学院）；BIH/MDC（柏林卫生研究院/马克斯·德尔布吕克分子医学中心）；University of Twente（特文特大学）；MPI-CBG（马克斯·普朗克分子细胞生物学与遗传学研究所）
We consider the task of pixel-wise semantic segmentation given a small set of labeled training images. Among two of the most popular techniques to address this task are Decision Forests (DF) and Neural Networks (NN). In this work, we explore the relationship between two special forms of these techniques: stacked DFs (namely Auto-context) and deep Convolutional Neural Networks (ConvNet). Our main contribution is to show that Auto-context can be mapped to a deep ConvNet with novel architecture, and thereby trained end-to-end. This mapping can be used as an initialization of a deep ConvNet, enabl…
Semantic image segmentation plays a pivotal role in many vision applications including autonomous driving and medical image analysis. Most of the former approaches move towards enhancing the performance in terms of accuracy with a little awareness of computational efficiency. In this paper, we introduce LiteSeg, a lightweight architecture for semantic image segmentation. In this work, we explore a new deeper version of Atrous Spatial Pyramid Pooling module (ASPP) and apply short and long residual connections, and depthwise separable convolution, resulting in a faster and efficient model. LiteS…
Scientific Computing and Imaging Institute, University of Utah（犹他大学科学计算与成像研究所）
We introduce a novel unsupervised loss function for learning semantic segmentation with deep convolutional neural nets (ConvNet) when densely labeled training images are not available. More specifically, the proposed loss function penalizes the L1-norm of the gradient of the label probability vector image , i.e. total variation, produced by the ConvNet. This can be seen as a regularization term that promotes piecewise smoothness of the label probability vector image produced by the ConvNet during learning. The unsupervised loss function is combined with a supervised loss in a semi-supervised s…
3D neural networks have become prevalent for many 3D vision tasks including object detection, segmentation, registration, and various perception tasks for 3D inputs. However, due to the sparsity and irregularity of 3D data, custom 3D operators or network designs have been the primary focus of research, while the size of networks or efficacy of parameters has been overlooked. In this work, we perform the first comprehensive study on the weight sparsity of spatially sparse 3D convolutional networks and propose a compact weight-sparse and spatially sparse 3D convnet (WS^3-Convnet) for semantic an…
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）
It is well believed that Transformer performs better in semantic segmentation compared to convolutional neural networks. Nevertheless, the original Vision Transformer may lack of inductive biases of local neighborhoods and possess a high time complexity. Recently, Swin Transformer sets a new record in various vision tasks by using hierarchical architecture and shifted windows while being more efficient. However, as Swin Transformer is specifically designed for image classification, it may achieve suboptimal performance on dense prediction-based segmentation task. Further, simply combing Swin T…
SwinMM: Masked Multi-view with Swin Transformers for 3D Medical Image Segmentation
Shanghai Jiao Tong University（上海交通大学）；University of Washington（华盛顿大学）；The Johns Hopkins University（约翰斯·霍普金斯大学）；University of Michigan, Ann Arbor（密歇根大学安娜堡分校）；University of California, Santa Cruz（加利福尼亚大学圣克鲁兹分校）；Tsinghua University（清华大学）；Stanford University（斯坦福大学）
SwinMamba: A hybrid local-global mamba framework for enhancing semantic segmentation of remotely sensed images
Xi'an Jiaotong-Liverpool University（西交利物浦大学）；University of Liverpool（利物浦大学）；Nanyang Technological University（南洋理工大学）；SJTUSME-COSMOPlat Joint Research Center for New Generation Industrial Intelligent Technology（上海交通大学机械与动力工程学院-卡奥斯新一代工业智能技术联合研究中心）
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Shenzhen Medical Biometrics Perception and Analysis Engineering Laboratory（深圳市医学生物特征感知与分析工程实验室）；Bio-Computing Research Center（生物计算研究中心）；Shenzhen Key Laboratory of Visual Object Detection and Recognition（深圳市视觉目标检测与识别重点实验室）
Automatic medical image segmentation has made great progress benefit from the development of deep learning. However, most existing methods are based on convolutional neural networks (CNNs), which fail to build long-range dependencies and global context connections due to the limitation of receptive field in convolution operation. Inspired by the success of Transformer in modeling the long-range contextual information, some researchers have expended considerable efforts in designing the robust variants of Transformer-based U-Net. Moreover, the patch division used in vision transformers usually …
University of California, Davis（加利福尼亚大学戴维斯分校）；NVIDIA（英伟达公司）
Aliasing refers to the phenomenon that high frequency signals degenerate into completely different ones after sampling. It arises as a problem in the context of deep learning as downsampling layers are widely adopted in deep architectures to reduce parameters and computation. The standard solution is to apply a low-pass filter (e.g., Gaussian blur) before downsampling. However, it can be suboptimal to apply the same filter across the entire content, as the frequency of feature maps can vary across both spatial locations and feature channels. To tackle this, we propose an adaptive content-aware…
Weakly Supervised Intracranial Hemorrhage Segmentation using Head-Wise Gradient-Infused Self-Attention Maps from a Swin Transformer in Categorical Learning
Technical Report for ICRA 2025 GOOSE 2D Semantic Segmentation Challenge: Leveraging Color Shift Correction, RoPE-Swin Backbone, and Quantile-based Label Denoising Strategy for Robust Outdoor Scene Understanding
Institute of Intelligent Systems（智能系统研究所）；College of Artificial Intelligence, National Yang Ming Chiao Tung University（人工智能学院，阳明交通大学）；Institute of Data Science, National Cheng Kung University（数据科学研究所，成功大学）
University of Science and Technology of China（中国科学技术大学）；Microsoft Research Asia（微软亚洲研究院）
The superior performance of Deformable Convolutional Networks arises from its ability to adapt to the geometric variations of objects. Through an examination of its adaptive behavior, we observe that while the spatial support for its neural features conforms more closely than regular ConvNets to object structure, this support may nevertheless extend well beyond the region of interest, causing features to be influenced by irrelevant image content. To address this problem, we present a reformulation of Deformable ConvNets that improves its ability to focus on pertinent image regions, through inc…
When Swin Transformer Meets KANs: An Improved Transformer Architecture for Medical Image Segmentation
Department of Computer Science and Engineering, University of Notre Dame（计算机科学与工程系，诺丁汉大学）；Department of Computer Science, University of Texas Rio Grande Valley（计算机科学系，德克萨斯大学里奥格兰德谷分校）
University of Texas at Austin（德克萨斯大学奥斯汀分校）；Eindhoven University of Technology（埃因霍温理工大学）；University of Jyväskylä（于韦斯屈莱大学）；University of Twente（特文特大学）；University of Luxembourg（卢森堡大学）
Transformers have quickly shined in the computer vision world since the emergence of Vision Transformers (ViTs). The dominant role of convolutional neural networks (CNNs) seems to be challenged by increasingly effective transformer-based models. Very recently, a couple of advanced convolutional models strike back with large kernels motivated by the local-window attention mechanism, showing appealing performance and efficiency. While one of them, i.e. RepLKNet, impressively manages to scale the kernel size to 31x31 with improved performance, the performance starts to saturate as the kernel size…
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
We propose iSegFormer, a memory-efficient transformer that combines a Swin transformer with a lightweight multilayer perceptron (MLP) decoder. With the efficient Swin transformer blocks for hierarchical self-attention and the simple MLP decoder for aggregating both local and global attention, iSegFormer learns powerful representations while achieving high computational efficiencies. Specifically, we apply iSegFormer to interactive 3D medical image segmentation.
PeLK: Parameter-efficient Large Kernel ConvNets with Peripheral Convolution
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Meituan（美团）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）
HST-MRF: Heterogeneous Swin Transformer with Multi-Receptive Field for Medical Image Segmentation
school of Mathematics and Statistics, Changsha University of Science and Technology（长沙理工大学数学与统计学院）；school of Computer and Communication Engineering, Changsha University of Science and Technology（长沙理工大学计算机与通信工程学院）
Semantic segmentation of brain tumors is a fundamental medical image analysis task involving multiple MRI imaging modalities that can assist clinicians in diagnosing the patient and successively studying the progression of the malignant entity. In recent years, Fully Convolutional Neural Networks (FCNNs) approaches have become the de facto standard for 3D medical image segmentation. The popular "U-shaped" network architecture has achieved state-of-the-art performance benchmarks on different 2D and 3D semantic segmentation tasks and across various imaging modalities. However, due to the limited…
Recently, vision transformers started to show impressive results which outperform large convolution based models significantly. However, in the area of small models for mobile or resource constrained devices, ConvNet still has its own advantages in both performance and model complexity. We propose ParC-Net, a pure ConvNet based backbone model that further strengthens these advantages by fusing the merits of vision transformers into ConvNets. Specifically, we propose position aware circular convolution (ParC), a light-weight convolution op which boasts a global receptive field while producing l…
SwinCross: Cross-modal Swin Transformer for Head-and-Neck Tumor Segmentation in PET/CT Images
Philips Research North America（飞利浦北美研究院）；Massachusetts General Hospital/Harvard Medical School（麻省总医院/哈佛医学院）；The Center for Advanced Medical Computing and Analysis, Massachusetts General Hospital/Harvard Medical School（麻省总医院/哈佛医学院高级医学计算与分析中心）
Tsinghua University（清华大学）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
We propose a dynamic filtering strategy with large sampling field for ConvNets (LS-DFN), where the position-specific kernels learn from not only the identical position but also multiple sampled neighbor regions. During sampling, residual learning is introduced to ease training and an attention mechanism is applied to fuse features from different samples. Such multiple samples enlarge the kernels' receptive fields significantly without requiring more parameters. While LS-DFN inherits the advantages of DFN, namely avoiding feature map blurring by position-wise kernels while keeping translation i…
Barlow-Swin: Toward a novel siamese-based segmentation architecture using Swin-Transformers
HSLU, Lucerne University of Applied Sciences and Arts（卢塞恩应用科学与艺术大学）；Technical University of Munich（慕尼黑技术大学）；University College London (UCL)（伦敦大学学院）；TOELT LLC AI lab（TOELT LLC人工智能实验室）
Technische Universität München（慕尼黑工业大学）；Fudan University（复旦大学）；Huawei Technologies（华为技术有限公司）
In the past few years, convolutional neural networks (CNNs) have achieved milestones in medical image analysis. Especially, the deep neural networks based on U-shaped architecture and skip-connections have been widely applied in a variety of medical image tasks. However, although CNN has achieved excellent performance, it cannot learn global and long-range semantic information interaction well due to the locality of the convolution operation. In this paper, we propose Swin-Unet, which is an Unet-like pure Transformer for medical image segmentation. The tokenized image patches are fed into the …
University of North Carolina at Charlotte（北卡罗来纳大学夏洛特分校）；Michigan State University（密歇根州立大学）
We propose the width-resolution mutual learning method (MutualNet) to train a network that is executable at dynamic resource constraints to achieve adaptive accuracy-efficiency trade-offs at runtime. Our method trains a cohort of sub-networks with different widths using different input resolutions to mutually learn multi-scale representations for each sub-network. It achieves consistently better ImageNet top-1 accuracy over the state-of-the-art adaptive network US-Net under different computation constraints, and outperforms the best compound scaled MobileNet in EfficientNet by 1.5%. The superi…
Xi’an University of Posts&Telecommunications（西安邮电大学）；Xidian University（西安电子科技大学）
The Resolution of feature maps is critical for medical image segmentation. Most of the existing Transformer-based networks for medical image segmentation are U-Net-like architecture that contains an encoder that utilizes a sequence of Transformer blocks to convert the input medical image from high-resolution representation into low-resolution feature maps and a decoder that gradually recovers the high-resolution representation from low-resolution feature maps. Unlike previous studies, in this paper, we utilize the network design style from the High-Resolution Network (HRNet), replace the convo…
Shahid Beheshti University（沙希德·贝赫什提大学）；RWTH Aachen University（亚琛工业大学）；Alzahra University（阿尔扎赫拉大学）；University of Regensburg（雷根斯堡大学）
Melanoma is caused by the abnormal growth of melanocytes in human skin. Like other cancers, this life-threatening skin cancer can be treated with early diagnosis. To support a diagnosis by automatic skin lesion segmentation, several Fully Convolutional Network (FCN) approaches, specifically the U-Net architecture, have been proposed. The U-Net model with a symmetrical architecture has exhibited superior performance in the segmentation task. However, the locality restriction of the convolutional operation incorporated in the U-Net architecture limits its performance in capturing long-range depe…
Deep Learning (DL) models are becoming larger, because the increase in model size might offer significant accuracy gain. To enable the training of large deep networks, data parallelism and model parallelism are two well-known approaches for parallel training. However, data parallelism does not help reduce memory footprint per device. In this work, we introduce Large deep 3D ConvNets with Automated Model Parallelism (LAMP) and investigate the impact of both input's and deep 3D ConvNets' size on segmentation accuracy. Through automated model parallelism, it is feasible to train large deep 3D Con…
School of Astronomy and Space Science, Nanjing University（南京大学天文与空间科学学院）；Institute of Science and Technology for Deep Space Exploration, Suzhou Campus, Nanjing University（南京大学苏州校区深空探测科学技术研究院）
Conservatoire National des Arts et Métiers（法国国立工艺学院）；IRCAD（消化外科研究所）；Sorbonne Université（索邦大学）；CNRS（法国国家科学研究中心）；ISIR（巴黎智能系统与机器人研究所）
Transformers have proved to be very effective for visual recognition tasks. In particular, vision transformers construct compressed global representations through self-attention and learnable class tokens. Multi-resolution transformers have shown recent successes in semantic segmentation but can only capture local interactions in high-resolution feature maps. This paper extends the notion of global tokens to build GLobal Attention Multi-resolution (GLAM) transformers. GLAM is a generic module that can be integrated into most existing transformer backbones. GLAM includes learnable global tokens…
Sano Centre for Computational Medicine（萨诺计算医学中心）；Informatics Institute, University of Amsterdam（阿姆斯特丹大学信息学研究所）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）
Cardiac Magnetic Resonance Imaging is commonly used for the assessment of the cardiac anatomy and function. The delineations of left and right ventricle blood pools and left ventricular myocardium are important for the diagnosis of cardiac diseases. Unfortunately, the movement of a patient during the CMR acquisition procedure may result in motion artifacts appearing in the final image. Such artifacts decrease the diagnostic quality of CMR images and force redoing of the procedure. In this paper, we present a Multi-task Swin UNEt TRansformer network for simultaneous solving of two tasks in the …
The computer-assisted radiologic informative report is currently emerging in dental practice to facilitate dental care and reduce time consumption in manual panoramic radiographic interpretation. However, the amount of dental radiographs for training is very limited, particularly from the point of view of deep learning. This study aims to utilize recent self-supervised learning methods like SimMIM and UM-MAE to increase the model efficiency and understanding of the limited number of dental radiographs. We use the Swin Transformer for teeth numbering, detection of dental restorations, and insta…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）
Edge detection has made significant progress with the help of deep Convolutional Networks (ConvNet). These ConvNet based edge detectors have approached human level performance on standard benchmarks. We provide a systematical study of these detectors' outputs. We show that the detection results did not accurately localize edge pixels, which can be adversarial for tasks that require crisp edge inputs. As a remedy, we propose a novel refinement architecture to address the challenging problem of learning a crisp edge detector using ConvNet. Our method leverages a top-down backward refinement path…
Enhanced Deformable Convolution with Center-invariant Offset and Edge-aware Mask
Beihang University（北京航空航天大学）；Shanghai Academy of Spaceflight Technology（上海航天技术研究院）；Shanghai Radio Equipment Research Institute（上海无线电设备研究所）；Cardiff University（卡迪夫大学）；Shenzhen University（深圳大学）；Sun Yat-sen University（中山大学）；Lancaster University（兰卡斯特大学）
The formidable accomplishment of Transformers in natural language processing has motivated the researchers in the computer vision community to build Vision Transformers. Compared with the Convolution Neural Networks (CNN), a Vision Transformer has a larger receptive field which is capable of characterizing the long-range dependencies. Nevertheless, the large receptive field of Vision Transformer is accompanied by the huge computational cost. To boost efficiency, the window-based Vision Transformers emerge. They crop an image into several local windows, and the self-attention is conducted withi…
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Disney Research（迪士尼研究院）；ETH Zurich（苏黎世联邦理工学院）
Inspired by recent advances of deep learning in instance segmentation and object tracking, we introduce video object segmentation problem as a concept of guided instance segmentation. Our model proceeds on a per-frame basis, guided by the output of the previous frame towards the object of interest in the next frame. We demonstrate that highly accurate object segmentation in videos can be enabled by using a convnet trained with static images only. The key ingredient of our approach is a combination of offline and online learning strategies, where the former serves to produce a refined mask from…
