U3M: Unbiased Multiscale Modal Fusion Model for Multimodal Semantic Segmentation
Institute of Artificial Intelligence (TeleAI), China Telecom（中国电信人工智能研究院（TeleAI））；School of Artificial Intelligence, OPtics and ElectroNics (iOPEN), Northwestern Polytechnical University（西北工业大学人工智能、光学与电子学院（iOPEN））
Bingyu Li, Da Zhang, Zhiyuan Zhao, Junyu Gao, Xuelong Li
Multimodal semantic segmentation is a pivotal component of computer vision and typically surpasses unimodal methods by utilizing rich information set from various sources.Current models frequently adopt modality-specific frameworks that inherently biases toward certain modalities. Although these biases might be advantageous in specific situations, they generally limit the adaptability of the models across different multimodal contexts, thereby potentially impairing performance. To address this issue, we leverage the inherent capabilities of the model itself to discover the optimal equilibrium in multimodal fusion and introduce U3M: An Unbiased Multiscale Modal Fusion Model for Multimodal Semantic Segmentation. Specifically, this method involves an unbiased integration of multimodal visual data. Additionally, we employ feature fusion at multiple scales to ensure the effective extraction and integration of both global and local features. Experimental results demonstrate that our approach achieves superior performance across multiple datasets, verifing its efficacy in enhancing the robustness and versatility of semantic segmentation in diverse settings. Our code is available at U3M-multimodal-semantic-segmentation.
Institute of Software Chinese Academy of Sciences（中国科学院软件研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Kitware（Kitware公司）；Nanjing Institute of Software Technology（南京软件技术研究院）
Image inpainting is an ill-posed problem to recover missing or damaged image content based on incomplete images with masks. Previous works usually predict the auxiliary structures (e.g., edges, segmentation and contours) to help fill visually realistic patches in a multi-stage fashion. However, imprecise auxiliary priors may yield biased inpainted results. Besides, it is time-consuming for some methods to be implemented by multiple stages of complex neural networks. To solve this issue, we develop an end-to-end multi-modality guided transformer network, including one inpainting branch and two …
COMO: Cross-Mamba Interaction and Offset-Guided Fusion for Multimodal Object Detection
School of Resource and Environmental Sciences, Wuhan University（武汉大学资源与环境科学学院）；State Key Laboratory of Information Engineering in Surveying, Mapping, and Remote Sensing, Wuhan University（武汉大学测绘遥感信息工程国家重点实验室）；School of Mathematics and Statistics, University of Glasgow（格拉斯哥大学数学与统计学院）；School of Geophysics and Geomatics, China University of Geoscience（中国地质大学（武汉）地球物理与空间信息学院）
University of Freiburg（弗莱堡大学）；Toyota Research Institute（丰田研究院）
Learning to reliably perceive and understand the scene is an integral enabler for robots to operate in the real-world. This problem is inherently challenging due to the multitude of object types as well as appearance changes caused by varying illumination and weather conditions. Leveraging complementary modalities can enable learning of semantically richer representations that are resilient to such perturbations. Despite the tremendous progress in recent years, most multimodal convolutional neural network approaches directly concatenate feature maps from individual modality streams rendering t…
The state-of-the-art models for medical image segmentation are variants of U-Net and fully convolutional networks (FCN). Despite their success, these models have two limitations: (1) their optimal depth is apriori unknown, requiring extensive architecture search or inefficient ensemble of models of varying depths; and (2) their skip connections impose an unnecessarily restrictive fusion scheme, forcing aggregation only at the same-scale feature maps of the encoder and decoder sub-networks. To overcome these two limitations, we propose UNet++, a new neural architecture for semantic and instance…
This article presents a multiscale patch based convolutional neural network for the automatic segmentation of brain tumors in multi-modality 3D MR images. We use multiscale deep supervision and inputs to train a convolutional network. We evaluate the effectiveness of the proposed approach on the BRATS 2017 segmentation challenge where we obtained dice scores of 0.755, 0.900, 0.782 and 95% Hausdorff distance of 3.63mm, 4.10mm, and 6.81mm for enhanced tumor core, whole tumor and tumor core respectively.
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；University of Chinese Academy of Sciences（中国科学院大学）
Remote sensing (RS) cross-modal text-image retrieval has attracted extensive attention for its advantages of flexible input and efficient query. However, traditional methods ignore the characteristics of multi-scale and redundant targets in RS image, leading to the degradation of retrieval accuracy. To cope with the problem of multi-scale scarcity and target redundancy in RS multimodal retrieval task, we come up with a novel asymmetric multimodal feature matching network (AMFMN). Our model adapts to multi-scale feature inputs, favors multi-source retrieval methods, and can dynamically filter r…
We share our recent findings in an attempt to train a universal segmentation network for various cell types and imaging modalities. Our method was built on the generalized U-Net architecture, which allows the evaluation of each component individually. We modified the traditional binary training targets to include three classes for direct instance segmentation. Detailed experiments were performed regarding training schemes, training settings, network backbones, and individual modules on the segmentation performance. Our proposed training scheme draws minibatches in turn from each dataset, and t…
Multiscale Structure-Guided Latent Diffusion for Multimodal MRI Translation
Computer Science and Engineering, Northeastern University, Shenyang, China（东北大学计算机科学与工程系，沈阳，中国）；Key Laboratory of Intelligent Computing in Medical Image of Ministry of Education, Northeastern University, Shenyang, China（教育部医学图像智能计算重点实验室，东北大学，沈阳，中国）；National Frontiers Science Center for Industrial Intelligence and Systems Optimization, Shenyang, China（工业智能与系统优化国家级前沿科学中心，沈阳，中国）；Amii, University of Alberta, Edmonton, Alberta, Canada（阿尔伯塔大学阿米人工智能研究所，埃德蒙顿，阿尔伯塔，加拿大）；AiShiWeiLai AI Research, Beijing, China（人工智能研究（北京）有限公司，北京，中国）
Alzheimer's Disease (AD) is a progressive neurodegenerative disease. Amnestic mild cognitive impairment (MCI) is a common first symptom before the conversion to clinical impairment where the individual becomes unable to perform activities of daily living independently. Although there is currently no treatment available, the earlier a conclusive diagnosis is made, the earlier the potential for interventions to delay or perhaps even prevent progression to full-blown AD. Neuroimaging scans acquired from MRI and metabolism images obtained by FDG-PET provide in-vivo view into the structure and func…
Learning multimodal representations involves integrating information from multiple heterogeneous sources of data. It is a challenging yet crucial area with numerous real-world applications in multimedia, affective computing, robotics, finance, human-computer interaction, and healthcare. Unfortunately, multimodal research has seen limited resources to study (1) generalization across domains and modalities, (2) complexity during training and inference, and (3) robustness to noisy and missing modalities. In order to accelerate progress towards understudied modalities and tasks while ensuring real…
Advanced Technology Development Centre（先进技术发展中心）；Department of Computer Science and Engineering（计算机科学与工程系）；Department of Electrical Engineering（电气工程系）；Indian Institute of Technology Kharagpur（印度理工学院卡哈拉格普尔分校）；India（印度（注：根据上下文，此处“India”为国家，非机构，原数据可能存在格式问题，按要求不提取国家信息，但按输出要求保留英文机构名））
Coronary artery disease accounts for a large number of deaths across the world and clinicians generally prefer using x-ray computed tomography or magnetic resonance imaging for localizing vascular pathologies. Interventional imaging modalities like intravascular ultrasound (IVUS) are used to adjunct diagnosis of atherosclerotic plaques in vessels, and help assess morphological state of the vessel and plaque, which play a significant role for treatment planning. Since speckle intensity in IVUS images are inherently stochastic in nature and challenge clinicians with accurate visibility of the ve…
Shuffle Mamba: State Space Models with Random Shuffle for Multi-Modal Image Fusion
University of Science and Technology of China（中国科学技术大学）；Institute of Intelligent Machines（智能机器研究所）；Hefei Institutes of Physical Science, Chinese Academy of Sciences（中国科学院合肥物质科学研究院）；Intelligent Agriculture Engineering Laboratory of Anhui Province, Institute of Intelligent Machines（安徽省智能农业工程实验室，智能机器研究所）
University of Electronic Science and Technology of China（电子科技大学）；Sichuan Cancer Hospital and Institute（四川省肿瘤医院·研究所）
Radiotherapy is the main treatment modality for nasopharynx cancer. Delineation of Gross Target Volume (GTV) from medical images such as CT and MRI images is a prerequisite for radiotherapy. As manual delineation is time-consuming and laborious, automatic segmentation of GTV has a potential to improve this process. Currently, most of the deep learning-based automatic delineation methods of GTV are mainly performed on medical images like CT images. However, it is challenged by the low contrast between the pathology regions and surrounding soft tissues, small target region, and anisotropic resol…
VLA-Mark: A cross modal watermark for large vision-language alignment model
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science and Technology（香港科技大学）；University of Toronto（多伦多大学）；Ant Group, Alibaba（蚂蚁集团，阿里巴巴）；New York University Shanghai（纽约大学上海分校）
UAVD-Mamba: Deformable Token Fusion Vision Mamba for Multimodal UAV Detection
College of Mechanical and Vehicle Engineering, Hunan University（湖南大学机械与车辆工程学院）；School of Electronic Information and Communications, Huazhong University of Science and Technology（华中科技大学电子信息与通信学院）
University of Magdeburg（马格德堡大学）；University Hospital Magdeburg（马格德堡大学医院）
Recent studies demonstrated the eligibility of convolutional neural networks (CNNs) for solving the image registration problem. CNNs enable faster transformation estimation and greater generalization capability needed for better support during medical interventions. Conventional fully-supervised training requires a lot of high-quality ground truth data such as voxel-to-voxel transformations, which typically are attained in a too tedious and error-prone manner. In our work, we use weakly-supervised learning, which optimizes the model indirectly only via segmentation masks that are a more access…
COMSATS University Islamabad（伊斯兰堡COMSATS大学）；Umeå University（于默奥大学）
U-Net is a widely adopted neural network in the domain of medical image segmentation. Despite its quick embracement by the medical imaging community, its performance suffers on complicated datasets. The problem can be ascribed to its simple feature extracting blocks: encoder/decoder, and the semantic gap between encoder and decoder. Variants of U-Net (such as R2U-Net) have been proposed to address the problem of simple feature extracting blocks by making the network deeper, but it does not deal with the semantic gap problem. On the other hand, another variant UNET++ deals with the semantic gap…
OSDM-MReg: Multimodal Image Registration based One Step Diffusion Model
Shanghai Key Laboratory of Intelligent Sensing and Recognition（上海智能感知与识别重点实验室）；Shanghai Jiao Tong University（上海交通大学）；Center of Digital Innovation（数字创新中心）
Macro2Micro: A Rapid and Precise Cross-modal Magnetic Resonance Imaging Synthesis using Multi-scale Structural Brain Similarity
Department of Brain and Cognitive Science, Seoul National University, Seoul, Republic of Korea（脑科学与认知科学系，首尔国立大学）；Department of Applied Bioengineering, Seoul National University, Seoul, Republic of Korea（应用生物工程系，首尔国立大学）；Department of Psychology, Seoul National University, Seoul, Republic of Korea（心理学系，首尔国立大学）；Interdisciplinary Program in Artificial Intelligence, Seoul National University, Seoul, Republic of Korea（人工智能跨学科项目，首尔国立大学）；Computational Science Initiative, Brookhaven National Laboratory, Upton, NY, USA（计算科学计划，布鲁赫斯国家实验室）；School of Economics, Sogang University（经济学院，成均馆大学）；Brookhaven National Laboratory（布鲁赫斯国家实验室）
Beijing University of Posts and Telecommunications（北京邮电大学）；Rutgers University（罗格斯大学）；Shanghai AI Laboratory（上海人工智能实验室）；Hasso Plattner Institute, University of Potsdam（波茨坦大学哈索普拉特纳研究所）；YITU Technology（依图科技）
Inspired by the success of BERT, several multimodal representation learning approaches have been proposed that jointly represent image and text. These approaches achieve superior performance by capturing high-level semantic information from large-scale multimodal pretraining. In particular, LXMERT and UNITER adopt visual region feature regression and label classification as pretext tasks. However, they tend to suffer from the problems of noisy labels and sparse semantic annotations, based on the visual features having been pretrained on a crowdsourced dataset with limited and inconsistent sema…
