Given the prominence of current 3D sensors, a fine-grained analysis on the basic point cloud data is worthy of further investigation. Particularly, real point cloud scenes can intuitively capture complex surroundings in the real world, but due to 3D data's raw nature, it is very challenging for machine perception. In this work, we concentrate on the essential visual task, semantic segmentation, for large-scale point cloud data collected in reality. On the one hand, to reduce the ambiguity in nearby points, we augment their local context by fully utilizing both geometric and semantic features in a bilateral structure. On the other hand, we comprehensively interpret the distinctness of the points from multiple resolutions and represent the feature map following an adaptive fusion method at point-level for accurate semantic segmentation. Further, we provide specific ablation studies and intuitive visualizations to validate our key modules. By comparing with state-of-the-art networks on three different benchmarks, we demonstrate the effectiveness of our network.
Huazhong University of Science & Technology（华中科技大学）；Peking University（北京大学）；Megvii Inc. (Face++)（旷视科技（Face++））
Semantic segmentation requires both rich spatial information and sizeable receptive field. However, modern approaches usually compromise spatial resolution to achieve real-time inference speed, which leads to poor performance. In this paper, we address this dilemma with a novel Bilateral Segmentation Network (BiSeNet). We first design a Spatial Path with a small stride to preserve the spatial information and generate high-resolution features. Meanwhile, a Context Path with a fast downsampling strategy is employed to obtain sufficient receptive field. On top of the two paths, we introduce a new…
Image-Conditioned Instance Prompt Network for Referring Remote Sensing Image Segmentation
School of Computer Science, Northwestern Polytechnical University, Xi'an, China（西北工业大学计算机科学学院，西安，中国）；College of Intelligent Robotics and Advanced Manufacturing, Fudan University, Shanghai, China（复旦大学智能机器人与先进制造学院，上海，中国）；Shenzhen Research Institute of Northwestern Polytechnical University, Shenzhen, China（西北工业大学深圳研究院，深圳，中国）
SRMF: A Data Augmentation and Multimodal Fusion Approach for Long-Tail UHR Satellite Image Segmentation
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Haina Institute of Zhejiang University, Advanced Technology Institute, Zhejiang University（浙江大学海纳研究院、先进技术研究院）；Binjiang Research Institute of Zhejiang University（浙江大学滨江研究院）
BEVANet: Bilateral Efficient Visual Attention Network for Real-Time Semantic Segmentation
Graduate Institute of Networking and Multimedia, National Taiwan University（网络与多媒体研究所，国立台湾大学）；Department of Computer Science and Information Engineering, National Taiwan University（计算机科学与信息工程系，国立台湾大学）
A Text-Image Fusion Method with Data Augmentation Capabilities for Referring Medical Image Segmentation
College of Information Science and Engineering（信息科学与工程学院）；Ritsumeikan University（立命馆大学）；Tiwaki Co., Ltd.（Tiwaki公司）；Dalian University of Technology（大连理工大学）；School of Software（软件学院）；Zhejiang University（浙江大学）
Beijing Institution of Technology（北京理工大学）；Baidu Inc.（百度公司）；Inception Institute of Artificial Intelligence（起源人工智能研究院）
Answering semantically-complicated questions according to an image is challenging in Visual Question Answering (VQA) task. Although the image can be well represented by deep learning, the question is always simply embedded and cannot well indicate its meaning. Besides, the visual and textual features have a gap for different modalities, it is difficult to align and utilize the cross-modality information. In this paper, we focus on these two problems and propose a Graph Matching Attention (GMA) network. Firstly, it not only builds graph for the image, but also constructs graph for the question …
National Engineering Laboratory for Video Technology - Shenzhen Division（视频技术国家工程实验室深圳分室）
Generic object detection algorithms have proven their excellent performance in recent years. However, object detection on underwater datasets is still less explored. In contrast to generic datasets, underwater images usually have color shift and low contrast; sediment would cause blurring in underwater images. In addition, underwater creatures often appear closely to each other on images due to their living habits. To address these issues, our work investigates augmentation policies to simulate overlapping, occluded and blurred objects, and we construct a model capable of achieving better gene…
The Chinese University of Hong Kong（香港中文大学）；The University of Hong Kong（香港大学）；MEGVII Technology（旷视科技）；SmartMore（思谋科技）
In this work, we present a conceptually simple yet effective framework for cross-modality 3D object detection, named voxel field fusion. The proposed approach aims to maintain cross-modality consistency by representing and fusing augmented image features as a ray in the voxel field. To this end, the learnable sampler is first designed to sample vital features from the image plane that are projected to the voxel grid in a point-to-ray manner, which maintains the consistency in feature representation with spatial context. In addition, ray-wise fusion is conducted to fuse features with the supple…
Current state-of-the-art detectors typically exploit feature pyramid to detect objects at different scales. Among them, FPN is one of the representative works that build a feature pyramid by multi-scale features summation. However, the design defects behind prevent the multi-scale features from being fully exploited. In this paper, we begin by first analyzing the design defects of feature pyramid in FPN, and then introduce a new feature pyramid architecture named AugFPN to address these problems. Specifically, AugFPN consists of three components: Consistent Supervision, Residual Feature Augmen…
University of Liverpool（利物浦大学）；Xi’an Jiaotong-Liverpool University (XJTLU)（西交利物浦大学）；Duke Kunshan University（昆山杜克大学）
Single-source domain generalization (SDG) in medical image segmentation is a challenging yet essential task as domain shifts are quite common among clinical image datasets. Previous attempts most conduct global-only/random augmentation. Their augmented samples are usually insufficient in diversity and informativeness, thus failing to cover the possible target domain distribution. In this paper, we rethink the data augmentation strategy for SDG in medical image segmentation. Motivated by the class-level representation invariance and style mutability of medical images, we hypothesize that unseen…
In this work, we propose a novel framework named Region-Aware Network (RANet), which learns the ability of anti-confusing in case of heavy occlusion, nearby person and symmetric appearance, for human pose estimation. Specifically, the proposed method addresses three key aspects, i.e., data augmentation, feature learning and prediction fusion, respectively. First, we propose Parsing-based Data Augmentation (PDA) to generate abundant data that synthesizes confusing textures. Second, we not only propose a Feature Pyramid Stem (FPS) to learn stronger low-level features in lower stage; but also inc…
University of Science and Technology of China（中国科学技术大学）
It has been well recognized that fusing the complementary information from depth-aware LiDAR point clouds and semantic-rich stereo images would benefit 3D object detection. Nevertheless, it is not trivial to explore the inherently unnatural interaction between sparse 3D points and dense 2D pixels. To ease this difficulty, the recent proposals generally project the 3D points onto the 2D image plane to sample the image data and then aggregate the data at the points. However, this approach often suffers from the mismatch between the resolution of point clouds and RGB images, leading to sub-optima…
Hanyang University（汉阳大学）；Phantom AI Inc.（Phantom AI公司）；Korea Electronics Technology Institute (KETI)（韩国电子技术研究院）
The goal of multi-modal learning is to use complimentary information on the relevant task provided by the multiple modalities to achieve reliable and robust performance. Recently, deep learning has led significant improvement in multi-modal learning by allowing for the information fusion in the intermediate feature levels. This paper addresses a problem of designing robust deep multi-modal learning architecture in the presence of imperfect modalities. We introduce deep fusion architecture for object detection which processes each modality using the separate convolutional neural network (CNN) a…
CMF-IoU: Multi-Stage Cross-Modal Fusion 3D Object Detection with IoU Joint Prediction
School of Automation and Intelligent Sensing & Institute of Image Processing and Pattern Recognition, Shanghai Jiao Tong University（自动化与智能感知学院及图像处理与模式识别研究所，上海交通大学）；School of Computing and Artificial Intelligence, Jiangxi University of Finance and Economics（计算机与人工智能学院，江西财经大学）；School of Automation and Intelligent Sensing & Institute of Image Processing and Pattern Recognition & Institute of Medical Robotics, Shanghai Jiao Tong University（自动化与智能感知学院及图像处理与模式识别研究所及医学机器人研究所，上海交通大学）
University of Chinese Academy of Sciences（中国科学院大学）；Damo Academy, Alibaba Group（阿里巴巴集团达摩院）
LiDAR semantic segmentation essential for advanced autonomous driving is required to be accurate, fast, and easy-deployed on mobile platforms. Previous point-based or sparse voxel-based methods are far away from real-time applications since time-consuming neighbor searching or sparse 3D convolution are employed. Recent 2D projection-based methods, including range view and multi-view fusion, can run in real time, but suffer from lower accuracy due to information loss during the 2D projection. Besides, to improve the performance, previous methods usually adopt test time augmentation (TTA), which…
Mississippi State University（密西西比州立大学）；University of Missouri-Columbia（密苏里大学哥伦比亚分校）
A significant challenge in object detection is accurate identification of an object's position in image space, whereas one algorithm with one set of parameters is usually not enough, and the fusion of multiple algorithms and/or parameters can lead to more robust results. Herein, a new computational intelligence fusion approach based on the dynamic analysis of agreement among object detection outputs is proposed. Furthermore, we propose an online versus just in training image augmentation strategy. Experiments comparing the results both with and without fusion are presented. We demonstrate that…
Opel Automobile GmbH（欧宝汽车公司）；Institute of Measurement and Control Systems, Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院测量与控制系统研究所）
3D pedestrian detection is a challenging task in automated driving because pedestrians are relatively small, frequently occluded and easily confused with narrow vertical objects. LiDAR and camera are two commonly used sensor modalities for this task, which should provide complementary information. Unexpectedly, LiDAR-only detection methods tend to outperform multisensor fusion methods in public benchmarks. Recently, PointPainting has been presented to eliminate this performance drop by effectively fusing the output of a semantic segmentation network instead of the raw image information. In thi…
Batch Augmentation with Unimodal Fine-tuning for Multimodal Fusion of Large Language Models
AI and Cyber Futures Institute, Charles Sturt University, Australia（人工智能与网络未来研究所，查尔斯·斯特劳特大学，澳大利亚）；Rural Health Research Institute, Charles Sturt University, Australia（农村健康研究研究所，查尔斯·斯特劳特大学，澳大利亚）；School of Computer Science and Engineering, Macau University of Science and Technology, Macao（计算机科学与工程学院，澳门科学技术大学，澳门）
针对超声图像与临床文本的多模态胎儿器官检测，提出单模态预训练、批次增强和图文特征融合方法，并在 FPU23 与 UPMC Food-101 上验证接近 SOTA 的效果。
Many-MobileNet: Multi-Model Augmentation for Robust Retinal Disease Classification
School of Computing and Augmented Intelligence, Arizona State University（亚利桑那州立大学计算与增强智能学院）；McKeley School of Engineering, Washington University in St. Louis（圣路易斯华盛顿大学麦凯尔工程学院）；School of Computing, Clemson University（克莱姆森大学计算学院）；Mayo Clinic（梅奥诊所）
Beyond Augmentation: Cross-Modal Transformer Fusion with Bi-directional Attention for Low-Data Aneurysm Screening
Carnegie Mellon University（卡内基梅隆大学）；Artificial Intelligence in Medicine (AIM) Program（医学人工智能（AIM）项目）；Mass General Brigham（马萨诸塞总医院）；Harvard Medical School（哈佛医学院）；Department of Radiation Oncology（放射肿瘤科）；Dana-Farber Cancer Institute（达纳-法伯癌症研究所）；Brigham and Women’s Hospital（布里洛妇产科医院）
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Iris segmentation is a deterministic part of the iris recognition system. Unreliable segmentation of iris regions especially the limbic area is still the bottleneck problem, which impedes more accurate recognition. To make further efforts on accurate and reliable iris segmentation, we propose a bilateral self-attention module and design Bilateral Transformer (BiTrans) with hierarchical architecture by exploring spatial and visual relationships. The bilateral self-attention module adopts a spatial branch to capture spatial contextual information without resolution reduction and a visual branch …
Brain lesions, including stroke and tumours, have a high degree of variability in terms of location, size, intensity and form, making automatic segmentation difficult. We propose an improvement to existing segmentation methods by exploiting the bilateral quasi-symmetry of healthy brains, which breaks down when lesions are present. Specifically, we use nonlinear registration of a neuroimage to a reflected version of itself ("reflective registration") to determine for each voxel its homologous (corresponding) voxel in the other hemisphere. A patch around the homologous voxel is added as a set of…
Zhejiang University（浙江大学）；University of Glasgow（格拉斯哥大学）
LiDAR and camera are two modalities available for 3D semantic segmentation in autonomous driving. The popular LiDAR-only methods severely suffer from inferior segmentation on small and distant objects due to insufficient laser points, while the robust multi-modal solution is under-explored, where we investigate three crucial inherent difficulties: modality heterogeneity, limited sensor field of view intersection, and multi-modal data augmentation. We propose a multi-modal 3D semantic segmentation model (MSeg3D) with joint intra-modal feature extraction and inter-modal feature fusion to mitigat…
Tracking tasks based on deep neural networks have greatly improved with the emergence of Siamese trackers. However, the appearance of targets often changes during tracking, which can reduce the robustness of the tracker when facing challenges such as aspect ratio change, occlusion, and scale variation. In addition, cluttered backgrounds can lead to multiple high response points in the response map, leading to incorrect target positioning. In this paper, we introduce two transformer-based modules to improve Siamese tracking called DASTSiam: the spatio-temporal (ST) fusion module and the Discrim…
Indian Institute of Technology, Varanasi（瓦拉纳西印度理工学院）；Nanyang Technological University（南洋理工大学）
This work summarizes our submission for the Task 3: Disease Classification of ISIC 2018 challenge in Skin Lesion Analysis Towards Melanoma Detection. We use a novel deep neural network (DNN) ensemble architecture introduced by us that can effectively classify skin lesions by using data-augmentation and bagging to address paucity of data and prevent over-fitting. The ensemble is composed of two DNN architectures: Inception-v4 and Inception-Resnet-v2. The DNN architectures are combined in to an ensemble by using a $1\times1$ convolution for fusion in a meta-learning layer.
For deep learning applications, the massive data development (e.g., collecting, labeling), which is an essential process in building practical applications, still incurs seriously high costs. In this work, we propose an effective data augmentation method based on generative adversarial networks (GANs), called Domain Fusion. Our key idea is to import the knowledge contained in an outer dataset to a target model by using a multi-domain learning GAN. The multi-domain learning GAN simultaneously learns the outer and target dataset and generates new samples for the target tasks. The simultaneous le…
CrossFuse: Learning Infrared and Visible Image Fusion by Cross-Sensor Top-K Vision Alignment and Beyond
School of Information Engineering, Guangdong University of Technology（广东工业大学信息工程学院）；CRRC Academy Co., Ltd.（中车研究院有限公司）；School of Computer Science, Sun Yat-sen University（中山大学计算机学院）
Cooperation Does Matter: Exploring Multi-Order Bilateral Relations for Audio-Visual Segmentation
School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)（中国科学院大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所）；Meituan（美团）
The Johns Hopkins University（约翰斯·霍普金斯大学）；The Johns Hopkins University School of Medicine（约翰斯·霍普金斯大学医学院）
There has been a debate on whether to use 2D or 3D deep neural networks for volumetric organ segmentation. Both 2D and 3D models have their advantages and disadvantages. In this paper, we present an alternative framework, which trains 2D networks on different viewpoints for segmentation, and builds a 3D Volumetric Fusion Net (VFN) to fuse the 2D segmentation results. VFN is relatively shallow and contains much fewer parameters than most 3D networks, making our framework more efficient at integrating 3D information for segmentation. We train and test the segmentation and fusion modules individu…
Humans can easily recognize actions with only a few examples given, while the existing video recognition models still heavily rely on the large-scale labeled data inputs. This observation has motivated an increasing interest in few-shot video action recognition, which aims at learning new actions with only very few labeled samples. In this paper, we propose a depth guided Adaptive Meta-Fusion Network for few-shot video recognition which is termed as AMeFu-Net. Concretely, we tackle the few-shot recognition problem from three aspects: firstly, we alleviate this extremely data-scarce problem by …
Deep Unfolding Multi-modal Image Fusion Network via Attribution Analysis
School of Mathematics and Statistics, Xi’an Jiaotong University（西安交通大学数学与统计学院）；ETH Zürich（苏黎世联邦理工学院）；School of Mathematics and Statistics, Northwestern Polytechnical University（西北工业大学数学与统计学院）
