Both high-level and high-resolution feature representations are of great importance in various visual understanding tasks. To acquire high-resolution feature maps with high-level semantic information, one common strategy is to adopt dilated convolutions in the backbone networks to extract high-resolution feature maps, such as the dilatedFCN-based methods for semantic segmentation. However, due to many convolution operations are conducted on the high-resolution feature maps, such methods have large computational complexity and memory consumption. In this paper, we propose one novel holistically-guided decoder which is introduced to obtain the high-resolution semantic-rich feature maps via the multi-scale features from the encoder. The decoding is achieved via novel holistic codeword generation and codeword assembly operations, which take advantages of both the high-level and low-level features from the encoder features. With the proposed holistically-guided decoder, we implement the EfficientFCN architecture for semantic segmentation and HGD-FPN for object detection and instance segmentation. The EfficientFCN achieves comparable or even better performance than state-of-the-art methods with only 1/3 of their computational costs for semantic segmentation on PASCAL Context, PASCAL VOC, ADE20K datasets. Meanwhile, the proposed HGD-FPN achieves $>2\%$ higher mean Average Precision (mAP) when integrated into several object detection frameworks with ResNet-50 encoding backbones.
CUHK-SenseTime Joint Laboratory, The Chinese University of Hong Kong（香港中文大学-商汤科技联合实验室）；Shenzhen Key Lab of Computer Vision and Pattern Recognition, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院计算机视觉与模式识别深圳市重点实验室）；SenseTime Research（商汤科技研究院）
Both performance and efficiency are important to semantic segmentation. State-of-the-art semantic segmentation algorithms are mostly based on dilated Fully Convolutional Networks (dilatedFCN), which adopt dilated convolutions in the backbone networks to extract high-resolution feature maps for achieving high-performance segmentation performance. However, due to many convolution operations are conducted on the high-resolution feature maps, such dilatedFCN-based methods result in large computational complexity and memory consumption. To balance the performance and efficiency, there also exist en…
Zhejiang University（浙江大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Inception Institute of Artificial Intelligence（Inception人工智能研究院）
In this paper, we present an Intersection-over-Union (IoU) guided two-stage 3D object detector with a voxel-to-point decoder. To preserve the necessary information from all raw points and maintain the high box recall in voxel based Region Proposal Network (RPN), we propose a residual voxel-to-point decoder to extract the point features in addition to the map-view features from the voxel based RPN. We use a 3D Region of Interest (RoI) alignment to crop and align the features with the proposal boxes for accurately perceiving the object position. The RoI-Aligned features are finally aggregated wi…
Semantic segmentation architectures are mainly built upon an encoder-decoder structure. These models perform subsequent downsampling operations in the encoder. Since operations on high-resolution activation maps are computationally expensive, usually the decoder produces output segmentation maps by upsampling with parameters-free operators like bilinear or nearest-neighbor. We propose a Neural Network named Guided Upsampling Network which consists of a multiresolution architecture that jointly exploits high-resolution and large context information. Then we introduce a new module named Guided U…
University of Science and Technology Beijing（北京科技大学）；University of Central Florida（中佛罗里达大学）；Scoop Medical（Scoop医疗）
The encoder-decoder architecture is widely used as a lightweight semantic segmentation network. However, it struggles with a limited performance compared to a well-designed Dilated-FCN model for two major problems. First, commonly used upsampling methods in the decoder such as interpolation and deconvolution suffer from a local receptive field, unable to encode global contexts. Second, low-level features may bring noises to the network decoder through skip connections for the inadequacy of semantic concepts in early encoder layers. To tackle these challenges, a Global Enhancement Method is pro…
GeoCueFormer: Geometry-Guided Wavelet Representation and Prediction-Cued Dual-Stage Decoder for Underwater Semantic Segmentation
Southwest University of Science and Technology（西南科技大学）；Columbia University（哥伦比亚大学）；Carnegie Mellon University（卡内基梅隆大学）；University of California, Berkeley（加州大学伯克利分校）
Existing studies on salient object detection (SOD) focus on extracting distinct objects with edge information and aggregating multi-level features to improve SOD performance. To achieve satisfactory performance, the methods employ refined edge information and low multi-level discrepancy. However, both performance gain and computational efficiency cannot be attained, which has motivated us to study the inefficiencies in existing encoder-decoder structures to avoid this trade-off. We propose TRACER, which detects salient objects with explicit edges by incorporating attention guided tracing modul…
School of Artificial Intelligence and School of Automation, Xi’an University of Posts and Telecommunications（人工智能学院和自动化学院，西安邮电大学）；Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University（电气电子工程系，香港理工大学）
Dual-Decoder Consistency via Pseudo-Labels Guided Data Augmentation for Semi-Supervised Medical Image Segmentation
Fuzhou University（福州大学）；Fujian Medical University Union Hospital（福建医科大学附属协和医院）；Fujian Medical Ultrasound Research Institute（福建省医学超声研究所）；Macao Polytechnic University（澳门理工大学）
SEG-SAM: Semantic-Guided SAM for Unified Medical Image Segmentation
School of Electronic and Information Engineering, South China University of Technology（华南理工大学电子与信息工程学院）；Pazhou Lab（琶洲实验室）；College of Electronic and Information Engineering, Tongji University（同济大学电子与信息工程学院）
The way features propagate in Fully Convolutional Networks is of momentous importance to capture multi-scale contexts for obtaining precise segmentation masks. This paper proposes a novel series-parallel hybrid paradigm called the Chained Context Aggregation Module (CAM) to diversify feature propagation. CAM gains features of various spatial scales through chain-connected ladder-style information flows and fuses them in a two-stage process, namely pre-fusion and re-fusion. The serial flow continuously increases receptive fields of output neurons and those in parallel encode different region-ba…
Frequency-domain Multi-modal Fusion for Language-guided Medical Image Segmentation
School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）；NLPR, MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；School of Artificial Intelligence, Anhui University（安徽大学人工智能学院）
Referring image segmentation aims to segment the image region of interest according to the given language expression, which is a typical multi-modal task. Existing methods either adopt the pixel classification-based or the learnable query-based framework for mask generation, both of which are insufficient to deal with various text-image pairs with a fix number of parametric prototypes. In this work, we propose an end-to-end framework built on transformer to perform Linguistic query-Guided mask generation, dubbed LGFormer. It views the linguistic features as query to generate a specialized prot…
Prompt-OVD is an efficient and effective framework for open-vocabulary object detection that utilizes class embeddings from CLIP as prompts, guiding the Transformer decoder to detect objects in both base and novel classes. Additionally, our novel RoI-based masked attention and RoI pruning techniques help leverage the zero-shot classification ability of the Vision Transformer-based CLIP, resulting in improved detection performance at minimal computational cost. Our experiments on the OV-COCO and OVLVIS datasets demonstrate that Prompt-OVD achieves an impressive 21.2 times faster inference speed…
University of Toronto Institute for Aerospace Studies（多伦多大学航空航天研究所）
Accurate and reliable 3D object detection is vital to safe autonomous driving. Despite recent developments, the performance gap between stereo-based methods and LiDAR-based methods is still considerable. Accurate depth estimation is crucial to the performance of stereo-based 3D object detection methods, particularly for those pixels associated with objects in the foreground. Moreover, stereo-based methods suffer from high variance in the depth estimation accuracy, which is often not considered in the object detection pipeline. To tackle these two issues, we propose CG-Stereo, a confidence-guid…
SGIFormer: Semantic-guided and Geometric-enhanced Interleaving Transformer for 3D Instance Segmentation
Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University（电子工程系，香港理工大学）；School of Mechanical Science and Engineering, Huazhong University of Science and Technology（机械科学与工程学院，华中科技大学）
Recent salient object detection (SOD) methods based on deep neural network have achieved remarkable performance. However, most of existing SOD models designed for low-resolution input perform poorly on high-resolution images due to the contradiction between the sampling depth and the receptive field size. Aiming at resolving this contradiction, we propose a novel one-stage framework called Pyramid Grafting Network (PGNet), using transformer and CNN backbone to extract features from different resolution images independently and then graft the features from transformer branch to CNN branch. An a…
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；University of Chinese Academy of Sciences（中国科学院大学）
Semantic segmentation of point clouds generates comprehensive understanding of scenes through densely predicting the category for each point. Due to the unicity of receptive field, semantic segmentation of point clouds remains challenging for the expression of multi-receptive field features, which brings about the misclassification of instances with similar spatial structures. In this paper, we propose a graph convolutional network DGFA-Net rooted in dilated graph feature aggregation (DGFA), guided by multi-basis aggregation loss (MALoss) calculated through Pyramid Decoders. To configure multi…
Co-Seg++: Mutual Prompt-Guided Collaborative Learning for Versatile Medical Segmentation
School of Computer Science, University of Nottingham Ningbo China（诺丁汉大学宁波校区计算机科学学院）；University of Nottingham, UK（诺丁汉大学（英国））；Graduate School of Information, Production and Systems, Waseda University, Japan（早稻田大学信息、生产与系统研究生院）；Yale University（耶鲁大学）
We develop a new edge detection algorithm that tackles two important issues in this long-standing vision problem: (1) holistic image training and prediction; and (2) multi-scale and multi-level feature learning. Our proposed method, holistically-nested edge detection (HED), performs image-to-image prediction by means of a deep learning model that leverages fully convolutional neural networks and deeply-supervised nets. HED automatically learns rich hierarchical representations (guided by deep supervision on side responses) that are important in order to approach the human ability resolve the c…
School of Computer Science and Technology, Zhejiang Normal University（浙江师范大学计算机科学与技术学院）；Metaverse and Artificial Intelligence Institute of Wenzhou University（温州大学元宇宙与人工智能研究院）
Multi-organ segmentation is a critical task in computer-aided diagnosis. While recent deep learning methods have achieved remarkable success in image segmentation, huge variations in organ size and shape challenge their effectiveness in multi-organ segmentation. To address these challenges, we propose a Spatial Prior-Guided Cross Dual Encoder Network (SPG-CDENet), a novel two-stage segmentation paradigm designed to improve multi-organ segmentation accuracy. Our SPG-CDENet consists of two key components: a spatial prior network and a cross dual encoder network. The prior network generates coars…
Harbin Institute of Technology（哈尔滨工业大学）；Nanyang Technological University（南洋理工大学）
Accurate segmentation of organs or lesions from medical images is crucial for reliable diagnosis of diseases and organ morphometry. In recent years, convolutional encoder-decoder solutions have achieved substantial progress in the field of automatic medical image segmentation. Due to the inherent bias in the convolution operations, prior models mainly focus on local visual cues formed by the neighboring pixels, but fail to fully model the long-range contextual dependencies. In this paper, we propose a novel Transformer-based Attention Guided Network called TransAttUnet, in which the multi-leve…
HER-Seg: Holistically Efficient Segmentation for High-Resolution Medical Images
School of Computer Science, University of Nottingham Ningbo China（诺丁汉大学宁波校区计算机科学学院）；University of Nottingham, UK（诺丁汉大学（英国））；Sichuan University Pittsburgh Institute, Sichuan University, China（四川大学匹兹堡研究院，四川大学，中国）；The Chinese University of Hong Kong, Hong Kong SAR（香港中文大学（香港特别行政区））；School of Computer Science, University of Nottingham, UK（诺丁汉大学（英国）计算机科学学院）；School of Computer Science, University of Lincoln, UK（林肯大学（英国）计算机科学学院）；HKISI, Chinese Academy of Sciences, Hong Kong SAR（香港科学院HKISI，香港特别行政区）
University of Southern California（南加州大学）；University of Minnesota（明尼苏达大学）；University of Colorado Boulder（科罗拉多大学博尔德分校）
The increasing availability and accessibility of numerous overhead images allows us to estimate and assess the spatial arrangement of groups of geospatial target objects, which can benefit many applications, such as traffic monitoring and agricultural monitoring. Spatial arrangement estimation is the process of identifying the areas which contain the desired objects in overhead images. Traditional supervised object detection approaches can estimate accurate spatial arrangement but require large amounts of bounding box annotations. Recent semi-supervised clustering approaches can reduce manual …
MGA-Net: A Novel Mask-Guided Attention Neural Network for Precision Neonatal Brain Imaging
Biomedical Research；Innovation Institute of C\'adiz (INiBICA) Research Unit, Puerta del Mar University, C\'adiz, Spain；Division of Neonatology, Department of Pediatrics, Puerta del Mar University Hospital, C\'adiz, Spain；Area of Pediatrics, Department of Child；Radiology, Medical School, University of C\'adiz, C\'adiz, Spain
A novel Encoder-Decoder Network with Guided Transmission Map (EDN-GTM) for single image dehazing scheme is proposed in this paper. The proposed EDN-GTM takes conventional RGB hazy image in conjunction with its transmission map estimated by adopting dark channel prior as the inputs of the network. The proposed EDN-GTM utilizes U-Net for image segmentation as the core network and utilizes various modifications including spatial pyramid pooling module and Swish activation to achieve state-of-the-art dehazing performance. Experiments on benchmark datasets show that the proposed EDN-GTM outperforms…
GLOBER: Coherent Non-autoregressive Video Generation via GLOBal Guided Video DecodER
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Generative modeling of high-dimensional data is a key problem in machine learning. Successful approaches include latent variable models and autoregressive models. The complementary strengths of these approaches, to model global and local image statistics respectively, suggest hybrid models that encode global image structure into latent variables while autoregressively modeling low level detail. Previous approaches to such hybrid models restrict the capacity of the autoregressive decoder to prevent degenerate models that ignore the latent variables and only rely on autoregressive modeling. Our …
RIKEN AIP（理化学研究所革新智能综合研究中心）；German Aerospace Center（德国航空航天中心）；Univ. Grenoble Alpes（格勒诺布尔-阿尔卑斯大学）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔理工学院）；GIPSA-lab（格勒诺布尔图像语音信号自动控制实验室）；The University of Tokyo（东京大学）
The fusion of input and guidance images that have a tradeoff in their information (e.g., hyperspectral and RGB image fusion or pansharpening) can be interpreted as one general problem. However, previous studies applied a task-specific handcrafted prior and did not address the problems with a unified approach. To address this limitation, in this study, we propose a guided deep decoder network as a general prior. The proposed network is composed of an encoder-decoder network that exploits multi-scale features of a guidance image and a deep decoder network that generates an output image. The two …
Depth guided any-to-any image relighting aims to generate a relit image from the original image and corresponding depth maps to match the illumination setting of the given guided image and its depth map. To the best of our knowledge, this task is a new challenge that has not been addressed in the previous literature. To address this issue, we propose a deep learning-based neural Single Stream Structure network called S3Net for depth guided image relighting. This network is an encoder-decoder model. We concatenate all images and corresponding depth maps as the input and feed them into the model…
We propose an optical flow-guided approach for semi-supervised video object segmentation. Optical flow is usually exploited as additional guidance information in unsupervised video object segmentation. However, its relevance in semi-supervised video object segmentation has not been fully explored. In this work, we follow an encoder-decoder approach to address the segmentation task. A model to extract the combined information from optical flow and the image is proposed, which is then used as input to the target model and the decoder network. Unlike previous methods where concatenation is used t…
Shandong Key Laboratory of Ubiquitous Intelligent Computing, University of Jinan（山东 ubiquitous 智能计算重点实验室，济南大学）；Department of Mechanical Engineering, Shandong University（山东大学机械工程学院）
针对逻辑异常检测难题，提出正态性先验引导的多语义融合网络，利用预训练视觉-语言模型和语义码本注入正常语义，引导异常重建趋近正常，在MVTec LOCO AD上达到SOTA。
Multi-task learning of dense prediction tasks, by sharing both the encoder and decoder, as opposed to sharing only the encoder, provides an attractive front to increase both accuracy and computational efficiency. When the tasks are similar, sharing the decoder serves as an additional inductive bias providing more room for tasks to share complementary information among themselves. However, increased sharing exposes more parameters to task interference which likely hinders both generalization and robustness. Effective ways to curb this interference while exploiting the inductive bias of sharing …
School of Computer Science and Engineering, Xi'an University of Technology（西安理工大学计算机科学与工程学院）
Image haze removal is highly desired for the application of computer vision. This paper proposes a novel Context Guided Generative Adversarial Network (CGGAN) for single image dehazing. Of which, an novel new encoder-decoder is employed as the generator. And it consists of a feature-extraction-net, a context-extractionnet, and a fusion-net in sequence. The feature extraction-net acts as a encoder, and is used for extracting haze features. The context-extraction net is a multi-scale parallel pyramid decoder, and is used for extracting the deep features of the encoder and generating coarse dehaz…
Permeability has a dominant influence on the flow properties of a natural fluid. Lattice Boltzmann simulator determines permeability from the nano and micropore network. The simulator holds millions of flow dynamics calculations with its accumulated errors and high consumption of computing power. To efficiently and consistently predict permeability, we propose a morphology decoder, a parallel and serial flow reconstruction of machine learning segmented heterogeneous Cretaceous texture from 3D micro computerized tomography and nuclear magnetic resonance images. For 3D vision, we introduce contr…
University of Duisburg-Essen（杜伊斯堡-埃森大学）；Ulm University（乌尔姆大学）；Technical University of Denmark（丹麦技术大学）
We present a lightweight encoder-decoder architecture for monocular depth estimation, specifically designed for embedded platforms. Our main contribution is the Guided Upsampling Block (GUB) for building the decoder of our model. Motivated by the concept of guided image filtering, GUB relies on the image to guide the decoder on upsampling the feature representation and the depth map reconstruction, achieving high resolution results with fine-grained details. Based on multiple GUBs, our model outperforms the related methods on the NYU Depth V2 dataset in terms of accuracy while delivering up to…
Memory-Guided Point Cloud Completion for Dental Reconstruction
College of Information and Intelligent Science, Donghua University, Shanghai, China（信息与智能科学学院，东华大学，上海，中国）；Institute of Artificial Intelligence, Shanghai University, Shanghai, China（人工智能研究院，上海大学，上海，中国）
Boosting Few-shot 3D Point Cloud Segmentation via Query-Guided Enhancement
Harbin Institute of Technology Shenzhen（哈尔滨工业大学深圳校区）；SmartMore（思谋科技）；Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies（广东省新型安全智能技术重点实验室）
Dense captioning in 3D point clouds is an emerging vision-and-language task involving object-level 3D scene understanding. Apart from coarse semantic class prediction and bounding box regression as in traditional 3D object detection, 3D dense captioning aims at producing a further and finer instance-level label of natural language description on visual appearance and spatial relations for each scene object of interest. To detect and describe objects in a scene, following the spirit of neural machine translation, we propose a transformer-based encoder-decoder architecture, namely SpaCap3D, to t…
AD-Net: Attention-based dilated convolutional residual network with guided decoder for robust skin lesion segmentation
COMSATS University Islamabad (CUI)（COMSATS大学伊斯兰堡校区）；University of Engineering and Technology (UET) Lahore（拉合尔工程技术大学）；University of New South Wales（新南威尔士大学）；IBADAT International University Islamabad（伊巴达特国际大学伊斯兰堡校区）
Attention-Guided Multi-scale Interaction Network for Face Super-Resolution
Institute of Advanced Technology, Nanjing University of Posts and Telecommunications（南京邮电大学先进技术研究所）；Key Laboratory of Artificial Intelligence, Ministry of Education（教育部人工智能重点实验室）；School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；School of Automation, Southeast University（东南大学自动化学院）；School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）
Northwestern Polytechnical University（西北工业大学）；National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology（国家空天地海一体化大数据应用技术国家工程实验室）；The University of Adelaide（阿德莱德大学）；MinSheng FinTech Corp. Ltd.（民生金融科技股份有限公司）
Reading irregular scene text of arbitrary shape in natural images is still a challenging problem, despite the progress made recently. Many existing approaches incorporate sophisticated network structures to handle various shapes, use extra annotations for stronger supervision, or employ hard-to-train recurrent neural networks for sequence modeling. In this work, we propose a simple yet strong approach for scene text recognition. With no need to convert input images to sequence representations, we directly connect two-dimensional CNN features to an attention-based sequence decoder which guided …
