Convolutional neural networks (CNNs) have been the de facto standard for nowadays 3D medical image segmentation. The convolutional operations used in these networks, however, inevitably have limitations in modeling the long-range dependency due to their inductive bias of locality and weight sharing. Although Transformer was born to address this issue, it suffers from extreme computational and spatial complexities in processing high-resolution 3D feature maps. In this paper, we propose a novel framework that efficiently bridges a {\bf Co}nvolutional neural network and a {\bf Tr}ansformer {\bf (CoTr)} for accurate 3D medical image segmentation. Under this framework, the CNN is constructed to extract feature representations and an efficient deformable Transformer (DeTrans) is built to model the long-range dependency on the extracted feature maps. Different from the vanilla Transformer which treats all image positions equally, our DeTrans pays attention only to a small set of key positions by introducing the deformable self-attention mechanism. Thus, the computational and spatial complexities of DeTrans have been greatly reduced, making it possible to process the multi-scale and high-resolution feature maps, which are usually of paramount importance for image segmentation. We conduct an extensive evaluation on the Multi-Atlas Labeling Beyond the Cranial Vault (BCV) dataset that covers 11 major human organs. The results indicate that our CoTr leads to a substantial performance improvement over other CNN-based, transformer-based, and hybrid methods on the 3D multi-organ segmentation task. Code is available at \def\UrlFont{\rm\small\ttfamily} \url{https://github.com/YtongXie/CoTr}
Purpose: Colorectal cancer (CRC) is the second most common cause of cancer mortality worldwide. Colonoscopy is a widely used technique for colon screening and polyp lesions diagnosis. Nevertheless, manual screening using colonoscopy suffers from a substantial miss rate of polyps and is an overwhelming burden for endoscopists. Computer-aided diagnosis (CAD) for polyp detection has the potential to reduce human error and human burden. However, current polyp detection methods based on object detection framework need many handcrafted pre-processing and post-processing operations or user guidance t…
Bridging the Performance Gap between DETR and R-CNN for Graphical Object Detection in Document Images
Technical University of Kaiserslautern（凯泽斯劳滕工业大学）；German Research Institute for Artificial Intelligence (DFKI)（德国人工智能研究中心）；Luleå University of Technology（吕勒奥理工大学）
Institute of Automation, CAS（中国科学院自动化研究所）；Microsoft Research Asia（微软亚洲研究院）
Existing object detection frameworks are usually built on a single format of object/part representation, i.e., anchor/proposal rectangle boxes in RetinaNet and Faster R-CNN, center points in FCOS and RepPoints, and corner points in CornerNet. While these different representations usually drive the frameworks to perform well in different aspects, e.g., better classification or finer localization, it is in general difficult to combine these representations in a single framework to make good use of each strength, due to the heterogeneous or non-grid feature extraction by different representations…
Conservatoire National des Arts et Métiers（法国国立工艺学院）；IRCAD（消化癌症研究所）
Transformer models achieve state-of-the-art results for image segmentation. However, achieving long-range attention, necessary to capture global context, with high-resolution 3D images is a fundamental challenge. This paper introduces the Full resolutIoN mEmory (FINE) transformer to overcome this issue. The core idea behind FINE is to learn memory tokens to indirectly model full range interactions while scaling well in both memory and computational costs. FINE introduces memory tokens at two levels: the first one allows full interaction between voxels within local image regions (patches), the …
Our goal is to design architectures that retain the groundbreaking performance of Convolutional Neural Networks (CNNs) for landmark localization and at the same time are lightweight, compact and suitable for applications with limited computational resources. To this end, we make the following contributions: (a) we are the first to study the effect of neural network binarization on localization tasks, namely human pose estimation and face alignment. We exhaustively evaluate various design choices, identify performance bottlenecks, and more importantly propose multiple orthogonal ways to boost p…
Diabetic retinopathy (DR) is a leading cause of vision loss among middle-aged and elderly people, which significantly impacts their daily lives and mental health. To improve the efficiency of clinical screening and enable the early detection of DR, a variety of automated DR diagnosis systems have been recently established based on convolutional neural network (CNN) or vision Transformer (ViT). However, due to the own shortages of CNN / ViT, the performance of existing methods using single-type backbone has reached a bottleneck. One potential way for the further improvements is integrating diff…
Given the dependency of current CNN architectures on a large training set, the possibility of using synthetic data is alluring as it allows generating a virtually infinite amount of labeled training data. However, producing such data is a non-trivial task as current CNN architectures are sensitive to the domain gap between real and synthetic data. We propose to adopt general-purpose GAN models for pixel-level image translation, allowing to formulate the domain gap itself as a learning problem. The obtained models are then used either during training or inference to bridge the domain gap. Here,…
Our goal is to design architectures that retain the groundbreaking performance of CNNs for landmark localization and at the same time are lightweight, compact and suitable for applications with limited computational resources. To this end, we make the following contributions: (a) we are the first to study the effect of neural network binarization on localization tasks, namely human pose estimation and face alignment. We exhaustively evaluate various design choices, identify performance bottlenecks, and more importantly propose multiple orthogonal ways to boost performance. (b) Based on our ana…
Vision Mamba in Remote Sensing: A Comprehensive Survey of Techniques, Applications and Outlook
School of Advanced Technology, Xi’an Jiaotong-Liverpool University（西安交通大学利物浦大学先进技术学院）；School of Electronics and Information Engineering, Beihang University（北航电子信息工程学院）；Department of Paediatrics, Cambridge University（剑桥大学儿科部）；School of Computing and Mathematical Sciences, University of Leicester（莱斯特大学计算与数学科学学院）；Department of Computing Science, Robert Gordon University（罗伯特·戈登大学计算机科学系）；National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所模式识别国家实验室）；Nanyang Technological University, Singapore（新加坡南洋理工大学）；Department of Computer Science, University of Liverpool（利物浦大学计算机科学系）
Bridging global context interactions correctly is important for high-fidelity image completion with large masks. Previous methods attempting this via deep or large receptive field (RF) convolutions cannot escape from the dominance of nearby interactions, which may be inferior. In this paper, we propose to treat image completion as a directionless sequence-to-sequence prediction task, and deploy a transformer to directly capture long-range dependence in the encoder. Crucially, we employ a restrictive CNN with small and non-overlapping RF for weighted token representation, which allows the trans…
Advances in Medical Image Segmentation: A Comprehensive Survey with a Focus on Lumbar Spine Applications
Center for Informatics Science, School of Information Technology and Computer Science, Nile University（信息科学中心，信息科技与计算机科学学院，尼罗大学）；Graduate School of Information Science, University of Hyogo（信息科学研究生院，兵库大学）；Advanced Medical Engineering Research Institute, University of Hyogo（先进医学工程研究所，兵库大学）
Deep Learning and Machine Learning -- Object Detection and Semantic Segmentation: From Theory to Applications
Aarhus University（阿arhus大学）；Indiana University（印第安纳大学）；Kyoto University（京都大学）；Emory University（埃默里大学）；Sichuan University（四川大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Georgia Institute of Technology（佐治亚理工学院）；University of Hawaii（夏威夷大学）；Xi’an Jiaotong-Liverpool University（西安交通大学利物浦大学）；Zhejiang University（浙江大学）；Purdue University（普渡大学）
City University of Hong Kong（香港城市大学）；Tongji University（同济大学）；Peking University Shenzhen Graduate School（北京大学深圳研究生院）；Peng Cheng Laboratory（鹏城实验室）；City University of Hong Kong Shenzhen Research Institute（香港城市大学深圳研究院）
Convolution neural networks (CNNs) have succeeded in compressive image sensing. However, due to the inductive bias of locality and weight sharing, the convolution operations demonstrate the intrinsic limitations in modeling the long-range dependency. Transformer, designed initially as a sequence-to-sequence model, excels at capturing global contexts due to the self-attention-based architectures even though it may be equipped with limited localization abilities. This paper proposes CSformer, a hybrid framework that integrates the advantages of leveraging both detailed spatial information from C…
Beihang University（北京航空航天大学）；Hefei Innovation Research Institute, Beihang University（北京航空航天大学合肥创新研究院）；Beijing Advanced Innovation Centre for Biomedical Engineering（北京生物医学工程先进创新中心）；Beijing Advanced Innovation Centre for Big Data-Based Precision Medicine（北京大数据精准医疗先进创新中心）
Recently, deep convolutional neural networks have achieved great success for medical image segmentation. However, unlike segmentation of natural images, most medical images such as MRI and CT are volumetric data. In order to make full use of volumetric information, 3D CNNs are widely used. However, 3D CNNs suffer from higher inference time and computation cost, which hinders their further clinical applications. Additionally, with the increased number of parameters, the risk of overfitting is higher, especially for medical images where data and annotations are expensive to acquire. To issue thi…
Both classical Fourier transform-based methods and neural network methods are widely used in image processing tasks. The former has better interpretability, whereas the latter often achieves better performance in practice. This paper introduces ButterflyNet2D, a regular CNN with sparse cross-channel connections. A Fourier initialization strategy for ButterflyNet2D is proposed to approximate Fourier transforms. Numerical experiments validate the accuracy of ButterflyNet2D approximating both the Fourier and the inverse Fourier transforms. Moreover, through four image processing tasks and image d…
University of Science and Technology of China（中国科学技术大学）
There still remains an extreme performance gap between Vision Transformers (ViTs) and Convolutional Neural Networks (CNNs) when training from scratch on small datasets, which is concluded to the lack of inductive bias. In this paper, we further consider this problem and point out two weaknesses of ViTs in inductive biases, that is, the spatial relevance and diverse channel representation. First, on spatial aspect, objects are locally compact and relevant, thus fine-grained feature needs to be extracted from a token and its neighbors. While the lack of data hinders ViTs to attend the spatial re…
Rethinking Plant Disease Diagnosis: Bridging the Academic-Practical Gap with Vision Transformers and Zero-Shot Learning
Department of Computer Science, Mohamed El Bachir El Ibrahimi University（计算机科学系，莫亨德·埃巴希里·埃布拉希米大学）；Laboratory of Physical Chemistry and Biological Materials（物理化学与生物材料实验室）；Laboratory of Informatics and its Applications of M’sila, Mohamed Boudiaf University（信息学及其应用实验室，莫赫梅德·布迪亚夫大学）；Department of Computer Science, Mohamed Boudiaf University（计算机科学系，莫赫梅德·布迪亚夫大学）；Department of agricultural science, Mohamed El Bachir El Ibrahimi University（农业科学系，莫亨德·埃巴希里·埃布拉希米大学）；Laboratoire d’Amélioration et de Développement de la Production Végétale et Animale (LADPVA), University of Ferhat ABBAS (UFAS-Sétif1)（植物和动物生产改良与发展实验室（LADPVA），费哈特·阿巴斯大学（UFAS-Sétif1））
Bridging visual saliency and large language models for explainable deep learning in medical imaging
Department of Mathematics and Computer science（数学与计算机科学系）；Institute of Cognitive Science（认知科学研究所）；Rhodes University（罗德斯大学）；National Institute for Theoretical and Computational Sciences (NITheCS)（理论与计算科学国家研究所）
MambaBack: Bridging Local Features and Global Contexts in Whole Slide Image Analysis
Nhu Department of Electrical Engineering and Computer Science, University of California, Irvine（加州大学尔湾分校Nhu电子工程与计算机科学系）；Department of Electrical & Computer Engineering, National University of Singapore（新加坡国立大学电子与计算机工程系）；PuzzleLogic Pte Ltd, Singapore（新加坡PuzzleLogic公司）
DuoFormer: Leveraging Hierarchical Representations by Local and Global Attention Vision Transformer
Scientific Computing and Imaging Institute, University of Utah（科学计算与成像研究所，犹他大学）；Kahlert School of Computing, University of Utah（计算学院，犹他大学）；Electrical and Computer Engineering, University of Utah（电气与计算机工程，犹他大学）；Department of Pathology, University of Utah（病理学系，犹他大学）
Beijing Jiaotong University（北京交通大学）；Hefei University of Technology（合肥工业大学）；Nanyang Technological University（南洋理工大学）
Convolutional Neural Network (CNN)-based image super-resolution (SR) has exhibited impressive success on known degraded low-resolution (LR) images. However, this type of approach is hard to hold its performance in practical scenarios when the degradation process is unknown. Despite existing blind SR methods proposed to solve this problem using blur kernel estimation, the perceptual quality and reconstruction accuracy are still unsatisfactory. In this paper, we analyze the degradation of a high-resolution (HR) image from image intrinsic components according to a degradation-based formulation mo…
VORTEX: Challenging CNNs at Texture Recognition by using Vision Transformers with Orderless and Randomized Token Encodings
São Carlos Institute of Physics, University of São Paulo（圣保罗大学圣卡洛斯物理研究所）；KTH Royal Institute of Technology（瑞典皇家理工学院）；São Paulo State University（圣保罗州立大学）
Beijing Laboratory of Intelligent Information Technology, School of Computer Science, Beijing Institute of Technology（北京理工大学计算机学院智能信息技术北京市重点实验室）
Recent studies have shown that aggregating convolutional features of a pre-trained Convolutional Neural Network (CNN) can obtain impressive performance for a variety of visual tasks. The symmetric Positive Definite (SPD) matrix becomes a powerful tool due to its remarkable ability to learn an appropriate statistic representation to characterize the underlying structure of visual features. In this paper, we propose to aggregate deep convolutional features into an SPD matrix representation through the SPD generation and the SPD transformation under an end-to-end deep network. To this end, severa…
Rensselaer Polytechnic Institute（伦斯勒理工学院）；United Imaging Intelligence（联影智能）；University of California Riverside（加州大学河滨分校）；Northeastern University（东北大学）
Recent advances in Convolutional Neural Network (CNN) model interpretability have led to impressive progress in visualizing and understanding model predictions. In particular, gradient-based visual attention methods have driven much recent effort in using visual attention maps as a means for visual explanations. A key problem, however, is these methods are designed for classification and categorization tasks, and their extension to explaining generative models, e.g. variational autoencoders (VAE) is not trivial. In this work, we take a step towards bridging this crucial gap, proposing the firs…
