Deep multi-modal aggregation network for MR image reconstruction with auxiliary modality
National Center for Artificial Intelligence (NCAI), SDAIA（沙特数据与人工智能局国家人工智能中心（NCAI））；The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））；Shenzhen Research Institute of Big Data（深圳大数据研究院）；Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳人工智能与机器人研究院）；Institute of High Performance Computing, Agency for Science, Technology and Research（科技研究局高性能计算研究所）
Magnetic resonance (MR) imaging produces detailed images of organs and tissues with better contrast, but it suffers from a long acquisition time, which makes the image quality vulnerable to say motion artifacts. Recently, many approaches have been developed to reconstruct full-sampled images from partially observed measurements to accelerate MR imaging. However, most approaches focused on reconstruction over a single modality, neglecting the discovery of correlation knowledge between the different modalities. Here we propose a Multi-modal Aggregation network for mR Image recOnstruction with auxiliary modality (MARIO), which is capable of discovering complementary representations from a fully sampled auxiliary modality, with which to hierarchically guide the reconstruction of a given target modality. This implies that our method can selectively aggregate multi-modal representations for better reconstruction, yielding comprehensive, multi-scale, multi-modal feature fusion. Extensive experiments on IXI and fastMRI datasets demonstrate the superiority of the proposed approach over state-of-the-art MR image reconstruction methods in removing artifacts.
University of Southern California（南加州大学）；Facebook AI
To reduce a model size but retain performance, we often rely on knowledge distillation (KD) which transfers knowledge from a large "teacher" model to a smaller "student" model. However, KD on multimodal datasets such as vision-language tasks is relatively unexplored, and digesting multimodal information is challenging since different modalities present different types of information. In this paper, we perform a large-scale empirical study to investigate the importance and effects of each modality in knowledge distillation. Furthermore, we introduce a multimodal knowledge distillation framework…
Despite the successes of deep neural networks on many challenging vision tasks, they often fail to generalize to new test domains that are not distributed identically to the training data. The domain adaptation becomes more challenging for cross-modality medical data with a notable domain shift. Given that specific annotated imaging modalities may not be accessible nor complete. Our proposed solution is based on the cross-modality synthesis of medical images to reduce the costly annotation burden by radiologists and bridge the domain gap in radiological images. We present a novel approach for …
Adversarial Versus Federated: An Adversarial Learning based Multi-Modality Cross-Domain Federated Medical Segmentation
Department of Electronics and Information Engineering, Beihang University（电子信息工程系，北航）；Department of Gynecology and Obstetrics, Beijing Shijitan Hospital, Capital Medical University（妇产科，北京什刹海医院，首都医科大学）；Department of Computer Science, University of Liverpool（计算机科学系，利物浦大学）；School of Computing and Mathematical Sciences, University of Leicester（计算与数学科学学院，莱斯特大学）
JDCNet: Confidence-Gated Privileged-Modality Distillation for Cost-Preserving X-ray Inference
Auckland University of Technology（奥克兰技术大学）；Guilin University of Electronic Technology（桂林电子科技大学）；Hikvision Technology Co., Ltd（海康威视技术有限公司）；Hebei University of Technology（河北工业大学）
ViKL: A Mammography Interpretation Framework via Multimodal Aggregation of Visual-knowledge-linguistic Features
Ningbo Institute of Life and Health Industry, UCAS（中国科学院大学宁波生命与健康产业研究院）；School of Automation Science and Engineering, South China University of Technology（华南理工大学自动化科学与工程学院）；University of Hong Kong（香港大学）
National University of Singapore（新加坡国立大学）；Sea AI Lab（Sea AI 实验室）
Domain adaptation (DA) aims to transfer knowledge from a label-rich but heterogeneous domain to a label-scare domain, which alleviates the labeling efforts and attracts considerable attention. Different from previous methods focusing on learning domain-invariant feature representations, some recent methods present generic semi-supervised learning (SSL) techniques and directly apply them to DA tasks, even achieving competitive performance. One of the most popular SSL techniques is pseudo-labeling that assigns pseudo labels for each unlabeled data via the classifier trained by labeled data. Howe…
Triple-Phase Multimodal Knowledge Aggregation Framework for Microbial Keratitis Subtype Diagnosis on Slit-Lamp Photography
Department of Biomedical Engineering, Duke University（杜克大学生物医学工程系）；Kellogg Eye Center, Department of Ophthalmology and Visual Sciences, University of Michigan（密歇根大学凯洛格眼科中心，眼科学与视觉科学系）；Department of Cornea and Refractive Surgery Services, Aravind Eye Care System（阿瓦因眼科医疗系统角膜与屈光手术部）；Bascom Palmer Eye Institute, Department of Ophthalmology, University of Miami Miller School of Medicine（迈阿密大学米勒医学院巴斯科姆·帕勒眼科研究所，眼科学系）；Flaum Eye Institute, Department of Ophthalmology, University of Rochester Medical Center（罗切斯特大学医学中心弗劳姆眼科研究所，眼科学系）；Department of Ophthalmology, Henry Ford Hospital（亨利福特医院眼科部）；Duke Eye Center, Duke University School of Medicine（杜克大学医学院杜克眼科中心）
Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Northwestern Polytechnical University（西北工业大学）；Terminus Group（特斯联集团）；Institute of High Performance Computing, A*STAR（新加坡科技研究局高性能计算研究所）
Accelerated multi-modal magnetic resonance (MR) imaging is a new and effective solution for fast MR imaging, providing superior performance in restoring the target modality from its undersampled counterpart with guidance from an auxiliary modality. However, existing works simply combine the auxiliary modality as prior information, lacking in-depth investigations on the potential mechanisms for fusing different modalities. Further, they usually rely on the convolutional neural networks (CNNs), which is limited by the intrinsic locality in capturing the long-distance dependency. To this end, we …
How to perform effective information fusion of different modalities is a core factor in boosting the performance of RGBT tracking. This paper presents a novel deep fusion algorithm based on the representations from an end-to-end trained convolutional neural network. To deploy the complementarity of features of all layers, we propose a recursive strategy to densely aggregate these features that yield robust representations of target objects in each modality. In different modalities, we propose to prune the densely aggregated features of all modalities in a collaborative way. In a specific, we e…
ClusMFL: A Cluster-Enhanced Framework for Modality-Incomplete Multimodal Federated Learning in Brain Imaging Analysis
School of Science and Engineering, the Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）理工学院）；Department of Computer Science and Engineering, Lehigh University（利哈伊大学计算机科学与工程系）；Department of Computer Science, Emory University（埃默里大学计算机科学系）
Due to the lack of properly annotated medical data, exploring the generalization capability of the deep model is becoming a public concern. Zero-shot learning (ZSL) has emerged in recent years to equip the deep model with the ability to recognize unseen classes. However, existing studies mainly focus on natural images, which utilize linguistic models to extract auxiliary information for ZSL. It is impractical to apply the natural image ZSL solutions directly to medical images, since the medical terminology is very domain-specific, and it is not easy to acquire linguistic models for the medical…
A Modality-Tailored Graph Modeling Framework for Urban Region Representation via Contrastive Learning
Center for Applied Statistics, School of Statistics, Innovation Platform, Renmin University of China（应用统计中心、统计学院、创新平台、中国人民大学）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学深圳学院）
Accelerated Multi-Contrast MRI Reconstruction via Frequency and Spatial Mutual Learning
University of Science and Technology of China（中国科学技术大学）；Stanford University（斯坦福大学）；Centre for Artificial Intelligence and Robotics (CAIR), HKISI-CAS（香港智能科学研究所-人工智能与机器人中心（CAIR））
Tianjin University（天津大学）；The University of Hong Kong（香港大学）；Huanhu Hospital（环湖医院）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science and Technology（香港科技大学）
Multi-modal MR imaging is routinely used in clinical practice to diagnose and investigate brain tumors by providing rich complementary information. Previous multi-modal MRI segmentation methods usually perform modal fusion by concatenating multi-modal MRIs at an early/middle stage of the network, which hardly explores non-linear dependencies between modalities. In this work, we propose a novel Nested Modality-Aware Transformer (NestedFormer) to explicitly explore the intra-modality and inter-modality relationships of multi-modal MRIs for brain tumor segmentation. Built on the transformer-based…
Johns Hopkins University（约翰斯·霍普金斯大学）；Johns Hopkins School of Medicine（约翰斯·霍普金斯大学医学院）；National Institute on Aging, National Institute of Health（美国国立卫生研究院国家老龄化研究所）
Disentangling anatomical and contrast information from medical images has gained attention recently, demonstrating benefits for various image analysis tasks. Current methods learn disentangled representations using either paired multi-modal images with the same underlying anatomy or auxiliary labels (e.g., manual delineations) to provide inductive bias for disentanglement. However, these requirements could significantly increase the time and cost in data collection and limit the applicability of these methods when such data are not available. Moreover, these methods generally do not guarantee …
Multi-modal Uncertainty Robust Tree Cover Segmentation For High-Resolution Remote Sensing Images
School of Information and Electronics, Beijing Institute of Technology（信息与电子学院，北京理工大学）；National Key Laboratory of Science and Technology on Space-Born Intelligent Information Processing（空间智能信息处理国家重点实验室）；Department of Electrical and Computer Engineering, University of Delaware（电气与计算机工程系，德雷塞尔大学）；Swiss Federal Institute for Forest, Snow, and Landscape Research WSL（瑞士森林、雪和景观研究联邦 institute WSL）
Image-based characterization and disease understanding involve integrative analysis of morphological, spatial, and topological information across biological scales. The development of graph convolutional networks (GCNs) has created the opportunity to address this information complexity via graph-driven architectures, since GCNs can perform feature aggregation, interaction, and reasoning with remarkable flexibility and efficiency. These GCNs capabilities have spawned a new wave of research in medical imaging analysis with the overarching goal of improving quantitative disease understanding, mon…
